Welcome to ChubaoFS(Chubao File System)

Introduction

ChubaoFS(Chubao File System) is a distributed file system that is designed to natively support large scale container platforms.

High Level Architecture

Architecture

ChubaoFS consists of a metadata subsystem, a data subsystem, and a resource manager, and can be accessed by different clients (as a set of application processes) hosted on the containers through different file system instances called volumes.

The metadata subsystem stores the file metadata, and consists of a set of meta nodes. Each meta node consists of a set of meta partitions.

The data subsystem stores the file contents, and consists of a set of data nodes. Each data node consists of a set of data partitions.

The volume is a logical concept in ChubaoFS and consists of one or multiple meta partitions and one or multiple data partitions. Each partition can only be assigned to a single volume. From a client’s perspective, the volume can be viewed as a file system instance that contains data accessible by the containers. A volume can be mounted to multiple containers so that files can be shared among different clients simultaneously, and needs to be created at the very beginning before the any file operation. A ChubaoFS cluster deployed at each data center can have hundreds of thousands of volumes, whose data sizes vary from a few gigabytes to several terabytes.

Generally speaking, the resource manager periodically communicates with the metadata subsystem and data subsystem to manage the meta nodes and data nodes, respectively. Each client periodically communicates with the resource manager to obtain the up-to-date view of the mounted volume. A file operation usually initiates the communications from the client to the corresponding meta node and data node directly, without the involvement of the resource manager. The updated view of the mounted volume, as well as the file metadata are usually cached at the client side to reduce the communication overhead.

Features

Scalable Meta Management

The metadata operations could make up as much as half of typical file system workloads. On our platform, this becomes even more important as there could be heavy accesses to the metadata of files by tens of thousands of clients simultaneously. A single node that stores the file metadata could easily become the performance bottleneck. As a result, we employ a distributed metadata subsystem to manage the file metadata. In this way, the metadata requests from different clients can be forwarded to different nodes, which improves the scalability of the entire system. The metadata subsystem can be considered as an in-memory datastore of the file metadata. It can have thousands of meta nodes, each of which can have hundreds of billions of meta partitions. Each meta partition on a meta node stores the file metadata in memory by maintaining a set of inodes and a set of dentries. We employ two b-trees called inodeTree and dentryTree for fast lookup of inodes and dentries in the memory. The inodeTree is indexed by the inode id, and the dentryTree is indexed by the dentry name and the parent inode id. We also maintain a range of the inode ids (denoted as start and end) stored on a meta partition for splitting.

General-Purpose Storage Engine

To reduce the storage cost, many applications and services are served from the same shared storage infrastructure (aka “multi-tenancy”). The workloads of different applications and services are mixed together, where the file size can vary from a few kilobytes to hundreds of gigabytes, and the files can be written in a sequential or random fashion. For example, the log files usually need to be written sequentially in the execution order of the code; some data analytics in the machine learning domain are based on the data stored on the underlying file system; and a database engine running on top of the file system can modify the stored data frequently. A dedicated file system needs to be able to serve for all these different workloads with excellent performance.

Strong Replication Consistency

E-commence venders who move their line of business applications to the cloud usually prefer strong consistency. For example, an image processing service may not want to provide the customer with an outdated image that does not match the product description. This can be easily achieved if there is only one copy of the file. But to ensure a distributed file system to continue operating properly in the event of machines failures, which can be caused by various reasons such as faulty hard drives, bad motherboards, etc, there are usually multiple replicas of the same file. As a result, in a desired file system, the data read from any of the replicas must be consistent with each other.

Relaxed POSIX Semantics and Metadata Atomicity

In a POSIX-compliant distributed file system, the behavior of serving multiple processes on multiple client nodes should be the same as the behavior of a local file system serving multiple processes on a single node with direct attached storage. ChubaoFS provides POSIX-compliant APIs. However, the POSIX consistency semantics, as well as the atomicity requirement between the inode and dentry of the same file, have been carefully relaxed in order to better align with the needs of applications and to improve the system performance.

Run Cluster by Yum Tools

Yum tools to run a ChubaoFS cluster for CentOS 7+ is provided. The list of RPM packages dependencies can be installed with:

$ yum install http://storage.jd.com/chubaofsrpm/latest/cfs-install-latest-el7.x86_64.rpm
$ cd /cfs/install
$ tree -L 2
.
├── install_cfs.yml
├── install.sh
├── iplist
├── src
└── template
    ├── client.json.j2
    ├── console.json.j2
    ├── create_vol.sh.j2
    ├── datanode.json.j2
    ├── grafana
    │   ├── grafana.ini
    │   ├── init.sh
    │   └── provisioning
    ├── master.json.j2
    ├── metanode.json.j2
    └── objectnode.json.j2

Set parameters of the ChubaoFS cluster in iplist.

  • [master], [datanode] , [metanode] , [console], [monitor] , [client] modules includes member IP addresses of each role.
  • [cfs:vars] module includes parameters for SSH connection. So make sure the port, username and password for SSH connection is unified before start.
  • #master config module includes parameters of Master.
Properties
Key Type Description Mandatory
master_clusterName string The cluster identifier Yes
master_listen string Http port which api service listen on Yes
master_prof string golang pprof port Yes
master_logDir string Path for log file storage Yes
master_logLevel string Level operation for logging. Default is info. No
master_retainLogs string the number of raft logs will be retain. Yes
master_walDir string Path for raft log file storage. Yes
master_storeDir string Path for RocksDB file storage,path must be exist Yes
master_exporterPort int The prometheus exporter port No
master_metaNodeReservedMem string If the metanode memory is below this value, it will be marked as read-only. Unit: byte. 1073741824 by default. No
  • #datanode config module includes parameters of DataNodes.
Properties
Key Type Description Mandatory
datanode_listen string Port of TCP network to be listen Yes
datanode_prof string Port of HTTP based prof and api service Yes
datanode_logDir string Path for log file storage Yes
datanode_logLevel string Level operation for logging. Default is info No
datanode_raftHeartbeat string Port of raft heartbeat TCP network to be listen Yes
datanode_raftReplica string Port of raft replicate TCP network to be listen Yes
datanode_raftDir string Path for raft log file storage No
datanode_exporterPort string Port for monitor system No
datanode_disks string slice
Format: PATH:RETAIN.
PATH: Disk mount point, must exists. RETAIN: Retain space. (Ranges: 20G-50G.)
RETAIN: The minimum free space(Bytes) reserved for the path.
 Yes
  • #metanode config module includes parameters of MetaNodes.
Properties
Key Type Description Mandatory  
metanode_listen string Listen and accept port of the server Yes  
metanode_prof string Pprof port Yes  
metanode_logLevel string Level operation for logging. Default is info No  
metanode_metadataDir string MetaNode store snapshot directory Yes  
metanode_logDir string Log directory Yes  
metanode_raftDir string Raft wal directory Yes  
metanode_raftHeartbeatPort string Raft heartbeat port Yes  
metanode_raftReplicaPort string Raft replicate port Yes  
metanode_exporterPort string Port for monitor system No  
metanode_totalMem string Max memory metadata used. The value needs to be higher than the value of metaNodeReservedMem in the master configuration. Unit: byte Yes  
  • #objectnode config module includes parameters of ObjectNodes.
Key Type Description Mandatory
objectnode_listen string Listen and accept port of the server Yes
objectnode_domains string slice
Domain of S3-like interface which makes wildcard domain support
Format: DOMAIN
 No
objectnode_logDir string Log directory Yes
objectnode_logLevel string
Level operation for logging.
Default: error
 No
objectnode_exporterPort string Port for monitor system No
objectnode_enableHTTPS string Enable HTTPS Yes
  • #console config module includes parameters of Console.
Properties
Key Type Description Mandatory
console_logDir string Path for log file storage Yes
console_logLevel string Level operation for logging. Default is info No
console_listen string Port of TCP network to be listen, default is 80 Yes
  • #client config module includes parameters of Client.
Properties
Key Type Description Mandatory
client_mountPoint string Mount point of the client Yes
client_volName string Volume name No
client_owner string Owner id Yes
client_SizeGB string Initial size to create the volume if it not exists No
client_logDir string Path for log file storage Yes
client_logLevel string Level operation for logging. Default is info No
client_exporterPort string Port exposed to monitor system Yes
client_profPort string Pprof port No 
[master]
10.196.59.198
10.196.59.199
10.196.59.200
[datanode]
...
[cfs:vars]
ansible_ssh_port=22
ansible_ssh_user=root
ansible_ssh_pass="password"
...
#master config
...
#datanode config
...
datanode_disks =  '"/data0:10737418240","/data1:10737418240"'
...
#metanode config
...
metanode_totalMem = "28589934592"
...
#objectnode config
...
#console config
...

For more configurations, please refer to Resource Manager (Master); Meta Subsystem; Data Subsystem; Client; Monitor; Console.

Start the resources of ChubaoFS cluster with script install.sh . (make sure the Master is started first)

$ bash install.sh -h
Usage: install.sh -r | --role [datanode | metanode | master | objectnode | console | monitor | client | all | createvol ]
$ bash install.sh -r master
$ bash install.sh -r metanode
$ bash install.sh -r datanode
$ bash install.sh -r objectnode
$ bash install.sh -r console
$ bash install.sh -r monitor
$ bash install.sh -r client

Check mount point at /cfs/mountpoint on client node defined in iplist .

Open http://consul.prometheus-cfs.local in browser for monitoring system(the IP of monitoring system is defined in iplist ).

Run Cluster Manually

Building

Use following command to build server and client:

make build

If the build is successful, cfs-server and cfs-client will be found in directory build/bin

Deployment

Start Resource Manager (Master)

nohup ./cfs-server -c master.json &

Sample master.json is shown as follows,

{
  "role": "master",
  "ip": "10.196.59.198",
  "listen": "17010",
  "prof":"17020",
  "id":"1",
  "peers": "1:10.196.59.198:17010,2:10.196.59.199:17010,3:10.196.59.200:17010",
  "retainLogs":"2000",
  "logDir": "/cfs/master/log",
  "logLevel":"info",
  "walDir":"/cfs/master/data/wal",
  "storeDir":"/cfs/master/data/store",
  "consulAddr": "http://consul.prometheus-cfs.local",
  "exporterPort": 9500,
  "clusterName":"chubaofs01",
  "metaNodeReservedMem": "1073741824"
}

For detailed explanations of master.json, please refer to Resource Manager (Master).

Start Metanode

nohup ./cfs-server -c meta.json &

Sample meta.json is shown as follows,

{
    "role": "metanode",
    "listen": "17210",
    "prof": "17220",
    "logLevel": "info",
    "metadataDir": "/cfs/metanode/data/meta",
    "logDir": "/cfs/metanode/log",
    "raftDir": "/cfs/metanode/data/raft",
    "raftHeartbeatPort": "17230",
    "raftReplicaPort": "17240",
    "totalMem":  "8589934592",
    "consulAddr": "http://consul.prometheus-cfs.local",
    "exporterPort": 9501,
    "masterAddr": [
        "10.196.59.198:17010",
        "10.196.59.199:17010",
        "10.196.59.200:17010"
    ]
}

For detailed explanations of meta.json, please refer to Meta Subsystem.

Start Datanode

  1. Prepare data directories

    Recommendation Using independent disks can reach better performance.

    Disk preparation

    1.1 Check available disks

    fdisk -l

    1.2 Build local Linux file system on the selected devices

    mkfs.xfs -f /dev/sdx

    1.3 Make mount point

    mkdir /data0

    1.4 Mount the device on mount point

    mount /dev/sdx /data0

  2. Start datanode

    nohup ./cfs-server -c datanode.json &

    Sample datanode.json is shown as follows,

    {
  "role": "datanode",
  "listen": "17310",
  "prof": "17320",
  "logDir": "/cfs/datanode/log",
  "raftDir": "/cfs/datanode/log",
  "logLevel": "info",
  "raftHeartbeat": "17330",
  "raftReplica": "17340",
  "consulAddr": "http://consul.prometheus-cfs.local",
  "exporterPort": 9502,
  "masterAddr": [
     "10.196.59.198:17010",
     "10.196.59.199:17010",
     "10.196.59.200:17010"
  ],
  "disks": [
     "/data0:10737418240",
     "/data1:10737418240"
  ]
}

For detailed explanations of datanode.json, please refer to Data Subsystem.

Start ObjectNode

nohup ./cfs-server -c objectnode.json &

Sample objectnode.json is shown as follows,

{
    "role": "objectnode",
    "domains": [
        "object.cfs.local"
    ],
    "listen": 17410,
    "masterAddr": [
       "10.196.59.198:17010",
       "10.196.59.199:17010",
       "10.196.59.200:17010"
    ],
    "logLevel": "info",
    "logDir": "/cfs/Logs/objectnode"
}

For detailed explanations of objectnode.json, please refer to Object Subsystem (ObjectNode).

Start Console

nohup ./cfs-server -c console.json &

Sample console.json is shown as follows,

{
    "role": "console",
    "logDir": "/cfs/log/",
    "logLevel": "debug",
    "listen": "80",
    "masterAddr": [
        "192.168.0.11:17010",
        "192.168.0.12:17010",
        "192.168.0.13:17010"
    ],
    "objectNodeDomain": "object.chubao.io",
    "monitor_addr": "http://192.168.0.102:9090",
    "dashboard_addr": "http://192.168.0.103",
    "monitor_app": "cfs",
    "monitor_cluster": "cfs"
}

For detailed explanations of console.json, please refer to Console.

Create Volume

By default, there are only a few data partitions allocated upon volume creation, and will be dynamically expanded according to actual usage.

curl -v "http://10.196.59.198:17010/admin/createVol?name=test&capacity=10000&owner=cfs"

For performance evaluation, extra data partitions shall be pre-created according to the amount of data nodes and disks to reach maximum performance.

curl -v "http://10.196.59.198:17010/dataPartition/create?name=test&count=120"

Mount Client

  1. Run modprobe fuse to insert FUSE kernel module.

  2. Run yum install -y fuse to install libfuse.

  3. Run cfs-client -c fuse.json to start a client daemon.

    Sample fuse.json is shown as follows,

    {
  "mountPoint": "/cfs/mountpoint",
  "volName": "ltptest",
  "owner": "ltptest",
  "masterAddr": "10.196.59.198:17010,10.196.59.199:17010,10.196.59.200:17010",
  "logDir": "/cfs/client/log",
  "profPort": "17510",
  "exporterPort": "9504",
  "logLevel": "info"
}

For detailed explanations of fuse.json, please refer to Client.

Note that end user can start more than one client on a single machine, as long as mountpoints are different.

Upgrading

  1. freeze the cluster
curl -v "http://10.196.59.198:17010/cluster/freeze?enable=true"
  1. upgrade each module
  2. closed freeze flag
curl -v "http://10.196.59.198:17010/cluster/freeze?enable=false"

Resource Manager (Master)

The resource manager manages the file system by processing different types of tasks, such as creating/deleting/updating/loading partitions and keeping track of the resource status (such as the memory/disk utilization). The resource manager is also responsible for creating new volumes and adding new meta/data nodes to the ChubaoFS cluster. It has multiple replicas, among which the consistency is maintained by a consensus algorithm such as Raft, and persisted to a key value store such as RocksDB for backup and recovery.

Utilization-Based Distribution/Placement

The resource manager is a utilization-based distribution strategy to places the file metadata and contents across different meta and data nodes. Because each node can have multiple partitions, and the partitions in a volume do not need to reside on the same node, this distribution can be controlled at a finer granularity, resulting a more efficient resource management. Specifically, the distribution of file metadata and contents works follows:

First, when mounting a volume, the client asks the resource manager for a set of available meta and data partitions. These partitions are usually the ones on the nodes with the lowest memory/disk utilizations. Later on, when writing a file, the client can only choose the meta and data partitions in a random fashion from the ones allocated by the resource manager.

Second, when the resource manager finds that all the partitions in a volume is about to be full, it automatically adds a set of new partitions to this volume. These partitions are usually the ones on the nodes with the lowest memory/disk utilizations. Note that, when a partition is full, or a threshold (i.e., the number of files on a meta partition or the number of extents on a data partition) is reached, no new data can be stored on this partition, although it can still be modified or deleted.

Replica Placement

When choosing partitions for the replicas, the resource manager ensures that two replicas of the same partition never reside on the same node.

Meta Partition Splitting

There is a special requirement when splitting a meta partition. In particular, if a meta partition is about to reach its upper limit of the number of stored inodes and dentries, a splitting task needs to be performed with the requirement to ensure that the inode ids stored at the newly created partition are unique from the ones stored at the original partition.

To meet this requirement, when splitting a meta partition, the resource manager cuts off the inode range of the meta partition in advance at a upper bound end, a value greater than highest inode id used so far (denoted as maxInodeID), and sends a split request to the meta node to (1) update the inode range from 1 to end for the original meta partition, and (2) create a new meta partition with the inode range from end + 1 to infinity for this volume. As a result, the inode range for these two meta partitions becomes [1, end] and [end + 1, infinity], respectively. If there is another file needs to be created, then its inode id will be chosen as maxInodeID + 1 in the original meta partition, or end + 1 in the newly created meta partition. The maxInodeID of each meta partition can be obtained by the periodical communication between the resource manager and the the meta nodes.

Exception Handling

When a request to a meta/data partition times out (e.g., due to network outage), the remaining replicas of this partition are marked as read-only. When a meta/data partition is no longer available (e.g., due to hardware failures), all the data on this partition will eventually be migrated to a new available partition manually. This unavailability is identified by the multiple failures reported by the node when operating the files.

Metadata Subsystem

The metadata operations could make up as much as half of typical file system workloads. this can be important as there could be heavy accesses to the metadata of files by tens of thousands of clients simultaneously. A single node that stores the file metadata could easily become the performance/storage bottleneck. As a result, we employ a distributed metadata subsystem to manage the file metadata. In this way, the metadata requests from different clients can be forwarded to different nodes, which improves the scalability of the entire system.

Internal Structure

The metadata subsystem can be considered as an in-memory datastore of the file metadata. It can have thousands of meta nodes, each of which can have a set of meta partitions. Each meta partition on a meta node stores the file metadata in memory by maintaining a set of inodes and a set of dentries.

Generally speaking, An inode is an object that represents the underlying file (or directory), and can be identified by an unsigned 64-bit integer called the inode id. A dentry is an object that represents the directory hierarchy and can be identified by a string name and the id of the parent inode. For example, if we have two directories foo and bar, where foo is the parent directory of bar, then there are two inodes: one for foo called i1, and the other for bar called i2, and one dentry to represent the hierarchy of these two directories where i2 is the current inode and i1 is the parent inode.

A meta partition can only store the inodes and dentries of the files from the same volume. We employ two b-trees called inodeTree and dentryTree for fast lookup of inodes and dentries in the memory. The inodeTree is indexed by the inode id, and the dentryTree is indexed by the dentry name and the parent inode id. We also maintain a range of the inode ids (denoted as start and end) stored on a meta partition for splitting (see Resource Manager (Master)).

Replication

The replication during file write is performed in terms of meta partitions. The replication consistency is ensured by a revision of the Raft consensus protocol called the MultiRaft, which has the advantage of reduced heartbeat network traffic comparing to the original version.

Failure Recovery

The in-memory meta partitions are persisted to the local disk by snapshots and logs for backup and recovery. Some techniques such as log compaction are used to reduce the log files sizes and shorten the recovery time.

It is worth noting that, a failure that happens during a metadata operation could result an orphan inode with which has no dentry to be associated. The memory and disk space occupied by this inode can be hard to free. To minimize the chance of this case to happen, the client always issues a retry after a failure until the request succeeds or the maximum retry limit is reached.

Data Subsystem

The data subsystem is optimized for the storage of large and small files, which can be accessed in a sequential or random fashion.

Data Subsystem Architecture

Features

  • Large File Storage

For large files, the contents are stored as a sequence of one or multiple extents, which can be distributed across different data partitions on different data nodes. Writing a new file to the extent store always causes the data to be written at the zero-offset of a new extent, which eliminates the need for the offset within the extent. The last extent of a file does not need to fill up its size limit by padding (i.e., the extent does not have holes), and never stores the data from other files.

  • Small File Storage

The contents of multiple small files are aggregated and stored in a single extent, and the physical offset of each file content in the extent is recorded in the corresponding meta node. ChubaoFS relies on the punch hole interface, textit{fallocate()}footnote{url{http://man7.org/linux/man-pages/man2/fallocate.2.html}}, to textit{asynchronous} free the disk space occupied by the to-be-deleted file. The advantage of this design is to eliminate the need of implementing a garbage collection mechanism and therefore avoid to employ a mapping from logical offset to physical offset in an extent~cite{haystack}. Note that this is different from deleting large files, where the extents of the file can be removed directly from the disk.

  • Replication

    The replication is performed in terms of partitions during file writes. Depending on the file write pattern, ChubaoFS adopts different replication strategies.

    When a file is sequentially written into ChubaoFS, a primary-backup replication protocol is used to ensure the strong consistency with optimized IO throughput.

    _images/workflow-sequential-write.png

    When overwriting an existing file portion during random writes, we employ a MultiRaft-based replication protocol, which is similar to the one used in the metadata subsystem, to ensure the strong consistency.

    _images/workflow-overwriting.png

  • Failure Recovery

    Because of the existence of two different replication protocols, when a failure on a replica is discovered, we first start the recovery process in the primary-backup-based replication by checking the length of each extent and making all extents aligned. Once this processed is finished, we then start the recovery process in our MultiRaft-based replication.

HTTP APIs

API Method Parameters Description
/disks GET N/A Get disk list and informations.
/partitions GET N/A Get parttion list and infomartions.
/partition GET partitionId[int] Get detail of specified partition.
/extent GET partitionId[int]&extentId[int] Get extent informations.
/stats GET N/A Get status of the datanode.

Object Subsystem (ObjectNode)

The object subsystem provides S3-compatible object storage interface. It makes possible for ChubaoFS becomes a fusion storage that expose two type interface (POSIX and S3-compatible). So that user can operate files stored in ChubaoFS by using native Amazon S3 SDKs.

Structure

_images/cfs-object-subsystem-structure.png

The ObjectNode is a functional subsystem node. It fetch volume view (volume topology) on demand from Resource Manager (Master). Each ObjectNode communicate with metadata subsystem (MetaNode) and data subsystem (DataNode) directly.

ObjectNode is stateless design with high scalability and it have ability to operate all files stored in the ChubaoFS cluster which it belong to directly without any volume-mount operation.

Features

  • Provides object storage interface compatible with native Amazon S3 SDKs.
  • Fusion storage expose two type interface (POSIX and S3-compatible).
  • Stateless and high scalability

Semantic Transform

Based on our POSIX-compatible design. Every file operate request comes from object storage interface need to be made semantic transform to POSIX.

POSIX Object Storage
Cluster Region
Volume Bucket
Path Key

Example:

_images/cfs-object-subsystem-semantic.png

Put object ‘example/a/b.txt’ will be create and write data to file ‘/a/b.txt’ in volume ‘example’.

User

Before using the object storage service, you need to create a user through the Master. While creating users, AccessKey and SecretKey will be generated for each user.

ChubaoFS uses the field Owner of the volume as the user ID. There are two ways to create users:

  • When creating a volume through the API of the Master, if there is no user with the same name as the owner of the volume in the cluster, a user with the user ID of Owner will be automatically created.
  • Create a user by calling the user management API of the Master.

Authentication

The signature validation algorithm in object storage interface is fully compatible with native Amazon S3 service. The authentication consisting of AccessKey and SecretKey generated by Resource Manager (Master) with user creation, which can be obtained through the Master API. The AccessKey is a 16-character string unique in the entire ChubaoFS cluster.

The user has all access permissions to the volume owned by him. Users can grant other users specified permissions to access volumes under their own names. The permissions are divided into the following three categories:

  • Readonly or readwrite permission.
  • Permission for a single operation, such as GetObject, PutObject, etc.
  • Custom permission.

When a user uses the object storage service to execute a certain operation, ChubaoFS will identify whether the user has the corresponding permission.

Invisible Temporary Data

In order to make write operation in object storage interface atomically. Every write operation will create and write data to an invisible temporary. The volume operator in ObjectNode puts file data into temporary which only have ‘inode’ without ‘dentry’ in metadata. When all the file data stored successfully, the volume operator create or update ‘dentry’ in metadata makes it visible to users.

Object Mode Conflict (Important)

The POSIX and object storage are two different types of storage product, and the object storage is a Key-Value pair storage service. So that the object with key ‘a/b/c’ and the object with key ‘a/b’ are different object without any conflict.

But ChubaoFS is based on POSIX design. According to semantic transformation rule, the ‘b’ part in key ‘a/b/c’ transform to folder ‘b’ under the folder ‘a’ , and in key ‘a/b’ transform to file ‘b’ under the folder ‘a’.

The object key like this is conflict in ChubaoFS.

When the object being operated has conflicts with the existing object due to the above mode inconsistency, the ObjectNode will return a 409 status code to the client.

Supported S3 Features

  • File object operations.
  • Directory object operations.
  • Multipart upload.
  • Parallel download for high-level SDK APIs.
  • Tagging for bucket and object.
  • User-defined metadata for object.
  • IP address and network segment black and white list for bucket ACL.
  • Signature Algorithm V2 and V4.
  • Cross-Origin Resource Sharing (CORS).

Unsupported S3 Features

  • Version
  • Restore deleted objects
  • Locking objects
  • Lifecycle configuration for bucket and object.
  • Hosting Websites
  • Encryption
  • BitTorrent

Supported APIs

API Reference
AbortMultipartUpload https://docs.aws.amazon.com/AmazonS3/latest/API/API_AbortMultipartUpload.html
CompleteMultipartUpload https://docs.aws.amazon.com/AmazonS3/latest/API/API_CompleteMultipartUpload.html
CopyObject https://docs.aws.amazon.com/AmazonS3/latest/API/API_CopyObject.html
CreateBucket https://docs.aws.amazon.com/AmazonS3/latest/API/API_CreateBucket.html
CreateMultipartUpload https://docs.aws.amazon.com/AmazonS3/latest/API/API_CreateMultipartUpload.html
DeleteBucket https://docs.aws.amazon.com/AmazonS3/latest/API/API_DeleteBucket.html
DeleteBucketCors https://docs.aws.amazon.com/AmazonS3/latest/API/API_DeleteBucketCors.html
DeleteBucketPolicy https://docs.aws.amazon.com/AmazonS3/latest/API/API_DeleteBucketPolicy.html
DeleteBucketTagging https://docs.aws.amazon.com/AmazonS3/latest/API/API_DeleteBucketTagging.html
DeleteObject https://docs.aws.amazon.com/AmazonS3/latest/API/API_DeleteObject.html
DeleteObjects https://docs.aws.amazon.com/AmazonS3/latest/API/API_DeleteObjects.html
DeleteObjectTagging https://docs.aws.amazon.com/AmazonS3/latest/API/API_DeleteObjectTagging.html
GetBucketAcl https://docs.aws.amazon.com/AmazonS3/latest/API/API_GetBucketAcl.html
GetBucketCors https://docs.aws.amazon.com/AmazonS3/latest/API/API_GetBucketCors.html
GetBucketLocation https://docs.aws.amazon.com/AmazonS3/latest/API/API_GetBucketLocation.html
GetBucketPolicy https://docs.aws.amazon.com/AmazonS3/latest/API/API_GetBucketPolicy.html
GetBucketTagging https://docs.aws.amazon.com/AmazonS3/latest/API/API_GetBucketTagging.html
GetObject https://docs.aws.amazon.com/AmazonS3/latest/API/API_GetObject.html
GetObjectAcl https://docs.aws.amazon.com/AmazonS3/latest/API/API_GetObjectAcl.html
GetObjectTagging https://docs.aws.amazon.com/AmazonS3/latest/API/API_GetObjectTagging.html
HeadBucket https://docs.aws.amazon.com/AmazonS3/latest/API/API_HeadBucket.html
HeadObject https://docs.aws.amazon.com/AmazonS3/latest/API/API_HeadObject.html
ListBuckets https://docs.aws.amazon.com/AmazonS3/latest/API/API_ListBuckets.html
ListMultipartUploads https://docs.aws.amazon.com/AmazonS3/latest/API/API_ListMultipartUploads.html
ListObjects https://docs.aws.amazon.com/AmazonS3/latest/API/API_ListObjects.html
ListObjectsV2 https://docs.aws.amazon.com/AmazonS3/latest/API/API_ListObjectsV2.html
ListParts https://docs.aws.amazon.com/AmazonS3/latest/API/API_ListParts.html
PutBucketAcl https://docs.aws.amazon.com/AmazonS3/latest/API/API_PutBucketAcl.html
PutBucketCors https://docs.aws.amazon.com/AmazonS3/latest/API/API_PutBucketCors.html
PutBucketPolicy https://docs.aws.amazon.com/AmazonS3/latest/API/API_PutBucketPolicy.html
PutBucketTagging https://docs.aws.amazon.com/AmazonS3/latest/API/API_PutBucketTagging.html
PutObject https://docs.aws.amazon.com/AmazonS3/latest/API/API_PutObject.html
PutObjectAcl https://docs.aws.amazon.com/AmazonS3/latest/API/API_PutObjectAcl.html
PutObjectTagging https://docs.aws.amazon.com/AmazonS3/latest/API/API_PutObjectTagging.html
UploadPart https://docs.aws.amazon.com/AmazonS3/latest/API/API_UploadPart.html

Supported SDKs

Object Node provides S3-compatible object storage interface, so that you can operate files by using native Amazon S3 SDKs.

Name Language Link
AWS SDK for Java Java https://aws.amazon.com/sdk-for-java/
AWS SDK for JavaScript JavaScript https://aws.amazon.com/sdk-for-browser/
AWS SDK for JavaScript in Node.js JavaScript https://aws.amazon.com/sdk-for-node-js/
AWS SDK for Go Go https://docs.aws.amazon.com/sdk-for-go/
AWS SDK for PHP PHP https://aws.amazon.com/sdk-for-php/
AWS SDK for Ruby Ruby https://aws.amazon.com/sdk-for-ruby/
AWS SDK for .NET .NET https://aws.amazon.com/sdk-for-net/
AWS SDK for C++ C++ https://aws.amazon.com/sdk-for-cpp/
Boto3 Python http://boto.cloudhackers.com

Client

The client runs on each container executing application code and exposes a file system interface to applications and can access a mounted volume via a user-space file system interface such as FUSE.

Client Caching

The client process runs entirely in the user space with its own cache, which has been used in the following cases.

To reduce the communication with the resource manager, the client caches the addresses of the available meta and data partitions assigned to the mounted volume, which can be obtained at the startup, and periodically synchronizes this available partitions with the resource manager.

To reduce the communication with the meta nodes, the client also caches the returned inodes and dentries when creating new files, as well as the data partition id, the extent id and the offset, after the file has been written to the data node successfully. When a file is opened for read/write, the client will force the cache metadata to be synchronous with the meta node.

To reduce the communication with the data nodes, the client caches the most recently identified leader. Our observation is that, when reading a file, the client may not know which data node is the current leader because the leader could change after a failure recovery. As a result, the client may try to send the read request to each replica one by one until a leader is identified. However, since the leader does not change frequently, by caching the last identified leader, the client can have minimized number of retries in most cases.

Integration with FUSE

The ChubaoFS client has been integrated with FUSE to provide a file system interface in the user space. In the past, low performance is considered the main disadvantage of such user-space file systems. But over the years, FUSE has made several improvement on its performance such as multithreading and write-back cache. In the future, we plan to develop our own POSIX-compliant file system interface in the kernel space to completely eliminate the overhead from FUSE.

Currently the write-back cache feature does not work well in ChubaoFS due to the following reason. The default write behavior of FUSE is called directIO, which bypasses the kernel’s page cache. This results in performance problems on writing small files as each write pushes the file data to the user daemon. The solution FUSE implemented was to make the page cache support a write-back policy that aggregates small data first, and then make writes asynchronous. With that change, file data can be pushed to the user daemon in larger blobs at a time. However, in real production, we found that the write-back cache is not very useful, because a write operation usually invoke another process that tries to balance the dirty pages (pages in the main memory that have been modified during writing data to disk are marked as “dirty” and have to be flushed to disk before they can be freed), which incurs extra overhead. This overhead becomes more obvious when small files are continuously written through FUSE.

Resource Manager (Master)

The cluster contains dataNodes,metaNodes,vols,dataPartitions and metaPartitions,they are managed by master server. The master server caches the metadata in mem,persist to GoLevelDB,and ensure consistence by raft protocol.

The master server manages dataPartition id to dataNode server mapping,metaPartition id to metaNode server mapping.

At lease 3 master nodes are required in respect to high availability.

Features

  • Multi-tenant, Resource Isolation
  • dataNodes,metaNodes shared,vol owns dataPartition and metaPartition exclusive
  • Asynchronous communication with dataNode and metaNode

Configurations

ChubaoFS use JSON as configuration file format.

Properties :header: “Key”, “Type”, “Description”, “Mandatory”
role string Role of process and must be set to master Yes
ip string host ip Yes
listen string Http port which api service listen on Yes
prof string golang pprof port Yes
id string identy different master node Yes
peers string the member information of raft group Yes
logDir string Path for log file storage Yes
logLevel string Level operation for logging. Default is error. No
retainLogs string the number of raft logs will be retain. Yes
walDir string Path for raft log file storage. Yes
storeDir string Path for RocksDB file storage,path must be exist Yes
clusterName string The cluster identifier Yes
exporterPort int The prometheus exporter port No
consulAddr string The consul register addr for prometheus exporter No
metaNodeReservedMem string If the metanode memory is below this value, it will be marked as read-only. Unit: byte. 1073741824 by default. No
heartbeatPort string Raft heartbeat port,5901 by default No
replicaPort string Raft replica Port,5902 by default No
nodeSetCap string the capacity of node set,18 by default No
missingDataPartitionInterval string how much time it has not received the heartbeat of replica,the replica is considered missing ,24 hours by default No
dataPartitionTimeOutSec string how much time it has not received the heartbeat of replica, the replica is considered not alive ,10 minutes by default No
numberOfDataPartitionsToLoad string the maximum number of partitions to check at a time,40 by default No
secondsToFreeDataPartitionAfterLoad string the task that release the memory occupied by loading data partition task can be start, only after secondsToFreeDataPartitionAfterLoad seconds ,300 by default No
tickInterval string the interval of timer which check heartbeat and election timeout,500 ms by default No
electionTick string how many times the tick timer has reset,the election is timeout,5 by default No

Example:

{
 "role": "master",
 "id":"1",
 "ip": "10.196.59.198",
 "listen": "17010",
 "prof":"17020",
 "peers": "1:10.196.59.198:17010,2:10.196.59.199:17010,3:10.196.59.200:17010",
 "retainLogs":"20000",
 "logDir": "/cfs/master/log",
 "logLevel":"info",
 "walDir":"/cfs/master/data/wal",
 "storeDir":"/cfs/master/data/store",
 "exporterPort": 9500,
 "consulAddr": "http://consul.prometheus-cfs.local",
 "clusterName":"chubaofs01",
 "metaNodeReservedMem": "1073741824"
}

Start Service

nohup ./master -c config.json > nohup.out &

Meta Subsystem

Metanode is the manager of meta partitions and replicated by MultiRaft. Each metanode manages various of partitions. Each partition covers an inode range, and maintains two in-memory btrees: inode btree and dentry btree.

At lease 3 meta nodes are required in respect to high availability.

Properties
Key Type Description Mandatory  
role string Role of process and must be set to metanode Yes  
listen string Listen and accept port of the server Yes  
prof string Pprof port Yes  
localIP string IP of network to be choose No. If not specified, the ip address used to communicate with the master is used.  
logLevel string Level operation for logging. Default is error No  
metadataDir string MetaNode store snapshot directory Yes  
logDir string Log directory Yes  
raftDir string Raft wal directory Yes  
raftHeartbeatPort string Raft heartbeat port Yes  
raftReplicaPort string Raft replicate port Yes  
consulAddr string Addresses of monitor system No  
exporterPort string Port for monitor system No  
masterAddr string Addresses of master server Yes  
zoneName string Specified zone. default by default. No  
totalMem string Max memory metadata used. The value needs to be higher than the value of metaNodeReservedMem in the master configuration. Unit: byte Yes  
deleteBatchCount int64 when deleting inodes, how many are deleted at a time ,500 by default No  

Example:

{
     "role": "metanode",
     "listen": "17210",
     "prof": "17220",
     "logLevel": "debug",
     "metadataDir": "/cfs/metanode/data/meta",
     "logDir": "/cfs/metanode/log",
     "raftDir": "/cfs/metanode/data/raft",
     "raftHeartbeatPort": "17230",
     "raftReplicaPort": "17240",
     "consulAddr": "http://consul.prometheus-cfs.local",
     "exporterPort": 9501,
     "totalMem":  "8589934592",
     "masterAddr": [
         "10.196.59.198:17010",
         "10.196.59.199:17010",
         "10.196.59.200:17010"
     ]
 }

Notice

  • listen, raftHeartbeatPort, raftReplicaPort can’t be modified after boot startup first time;
  • Above config would be stored under directory raftDir in constcfg file. If need modified forcely，you must delete this file manually;
  • These configuration items associated with master’s metanode infomation . If they have been modified, master would’t be found old metanode;

Data Subsystem

How To Start DataNode

Start a DataNode process by execute the server binary of ChubaoFS you built with -c argument and specify configuration file. At least 4 data nodes are required in respect to high availability.

nohup cfs-server -c datanode.json &

Configurations

Properties
Key Type Description Mandatory
role string Role of process and must be set to datanode Yes
listen string Port of TCP network to be listen Yes
localIP string IP of network to be choose No,If not specified, the ip address used to communicate with the master is used.
prof string Port of HTTP based prof and api service Yes
logDir string Path for log file storage Yes
logLevel string Level operation for logging. Default is error No
raftHeartbeat string Port of raft heartbeat TCP network to be listen Yes
raftReplica string Port of raft replicate TCP network to be listen Yes
raftDir string Path for raft log file storage No
consulAddr string Addresses of monitor system No
exporterPort string Port for monitor system No
masterAddr string slice Addresses of master server Yes
zoneName string Specified zone. default by default. No
disks string slice
Format: PATH:RETAIN.
PATH: Disk mount point. RETAIN: Retain space. (Ranges: 20G-50G.)
 Yes

Example:

{
    "role": "datanode",
    "listen": "17310",
    "prof": "17320",
    "logDir": "/cfs/datanode/log",
    "logLevel": "info",
    "raftHeartbeat": "17330",
    "raftReplica": "17340",
    "raftDir": "/cfs/datanode/log",
    "consulAddr": "http://consul.prometheus-cfs.local",
    "exporterPort": 9502,
    "masterAddr": [
         "10.196.59.198:17010",
         "10.196.59.199:17010",
         "10.196.59.200:17010"
    ],
     "disks": [
        "/data0:10737418240",
        "/data1:10737418240"
    ]
}

Notice

  • Since datanode uses SEEK_HOLE and SEEK_DATA operations which is supported by XFS (since Linux 3.5) and ext4 (since Linux 3.8), users should pay attention to the Linux kernel version on which datanodes are deployed.
  • listen, raftHeartbeat, raftReplica can’t be modified after boot startup first time.
  • Above config would be stored under directory raftDir in constcfg file. If need modified forcely, you must delete this file manually.
  • These configuration items associated with master’s datanode infomation. If they have been modified, master would’t be found old datanode.

Object Subsystem (ObjectNode)

How To start ObjectNode

Start a ObjectNode process by execute the server binary of ChubaoFS you built with -c argument and specify configuration file.

nohup cfs-server -c objectnode.json &

Note: If you do not intend to use the object storage service, you do not need to start the ObjectNode.

Configurations

Object Node using JSON format configuration file.

Properties

Key Type Description Mandatory
role string Role of process and must be set to objectnode Yes
listen string Listen and accept port of the server Yes
domains string slice
Domain of S3-like interface which makes wildcard domain support
Format: DOMAIN
 No
logDir string Log directory Yes
logLevel string
Level operation for logging.
Default: error
 No
masterAddr string slice
Format: HOST:PORT.
HOST: Hostname, domain or IP address of master (resource manager).
PORT: port number which listened by this master
 Yes
authNodes string slice
Format: HOST:PORT.
HOST: Hostname, domain or IP address of AuthNode.
PORT: port number which listened by this AuthNode
 Yes
exporterPort string Port for monitor system No
prof string Pprof port Yes

Example:

{
     "role": "objectnode",
     "listen": "17410",
     "domains": [
         "object.cfs.local"
     ],
     "logDir": "/cfs/Logs/objectnode",
     "logLevel": "info",
     "masterAddr": [
         "10.196.59.198:17010",
         "10.196.59.199:17010",
         "10.196.59.200:17010"

     ],
     "exporterPort": 9503,
     "prof": "7013"
}

Fetch Authentication Keys

First, you need create a user through User API, and then get keys information. Refer to User.

You can also use Command Line Interface (CLI) tool to create and get user’s AccessKey and SecretKey:

$ cli user create [USER_ID]
$ cli user info [USER_ID]

Using Object Storage Interface

Object Subsystem (ObjectNode) provides S3-compatible object storage interface, so that you can operate files by using native Amazon S3 SDKs.

For detail about list of supported APIs, see Supported S3-compatible APIs at Object Subsystem (ObjectNode)

For detail about list of supported SDKs, see Supported SDKs at Object Subsystem (ObjectNode)

Using S3cmd

Use s3cmd to access the ObjectNode deployed locally.

Installation

Install the s3cmd from https://s3tools.org/s3cmd .

Configuration

Edit s3cmd configuration file $HOME/.s3cfg

host_base = 127.0.0.1
host_bucket = 127.0.0.1
use_https = False
access_key = YOUR_ACCESS_KEY
secret_key = YOUR_SECRET_KEY

Example: making a bucket (volume)

s3cmd mb s3://my_volume
Bucket 's3://my_volume/' created

Example: uploading an local file to ChubaoFS

s3cmd put data.dat s3://my_volume
upload: 'data.dat' -> 's3://my_volume/data.dat'

Example: listing buckets (volumes)

s3cmd ls
2020-05-10 15:29 s3://my_volume

Example: listing files stored in a ChubaoFS volume

s3cmd ls s3://my_volume
                         DIR    s3://my_volume/backup/
2020-05-10  15:31   10485760    s3://my_volume/data.dat
2020005-10  15:33         10    s3://my_volume/hello.txt

Example: deleting file stored in a ChubaoFS volume

s3cmd rm s3://my_volume/data.dat
delete: 's3://my_volume/data.dat'

Detail usage for s3cmd see https://s3tools.org/usage .

Using AWS Java SDK

Use AWS Java SDK to access the ObjectNode deployed locally.

Install by Maven:

<dependency>
    <groupId>software.amazon.awssdk</groupId>
    <artifactId>s3</artifactId>
    <version>2.10.71</version>
</dependency>

Example: uploading file to ChubaoFS volume (PutObject)

Regions clientRegion = Region.of("*** Region name ***"); // Setup region (the cluster name)
String endpoint = "http://127.0.0.1"; // Setup endpoint
String accessKey = "*** Access Key ***"; // Setup AccessKey
String secretKey = "*** Secret Key ***"; // Setup SecretKey
String bucketName = "*** Bucket name ***"; // Setup target bucket (ChubaoFS Volume)
String objectKey = "*** File object key name ***"; // Setup object key
[]byte data = "*** Example File Data as String **".getBytes();

try {
    // Setup credential
    AwsCredentialsProvider credentialsProvider = StaticCredentialsProvider.create(AwsBasicCredentials.create(accessKey, secretKey));

    // Init S3 client
    S3Configuration configuration = S3Configuration.builder()
        .chunkedEncodingEnabled(true)
        .build();
    S3Client client = S3Client.builder()
        .region(region)
        .credentialsProvider(credentialsProvider)
        .endpointOverride(URI.create(endpoint))
        .serviceConfiguration(configuration)
        .build();

    // Upload file
    PutObjectRequest request = PutObjectRequest.builder()
        .bucket(bucketName)
        .key(objectKey)
        .build();
    RequestBody body = RequestBody.fromBytes(data);
    s3Client.putObject(request, body)
} catch (Exception e) {
    e.printStackTrace();
}

Example: downloading file stored in ChubaoFS volume (GetObject)

Regions clientRegion = Region.of("*** Region name ***"); // Setup region
String endpoint = "http://127.0.0.1"; // Setup endpoint
String accessKey = "*** Access Key ***"; // Setup AccessKey
String secretKey = "*** Secret Key ***"; // Setup SecretKey
String bucketName = "*** Bucket name ***"; // Setup target bucket (ChubaoFS Volume)
String objectKey = "*** File object key name ***"; // Setup object key

try {
    // Setup credential
    AwsCredentialsProvider credentialsProvider = StaticCredentialsProvider.create(AwsBasicCredentials.create(accessKey, secretKey));

    // Init S3 client
    S3Configuration configuration = S3Configuration.builder()
        .chunkedEncodingEnabled(true)
        .build();
    S3Client client = S3Client.builder()
        .region(region)
        .credentialsProvider(credentialsProvider)
        .endpointOverride(URI.create(endpoint))
        .serviceConfiguration(configuration)
        .build();

    // Get file data
    GetObjectRequest request = GetObjectRequest.builder()
        .bucket(bucketName)
        .key(objectKey)
        .build();

    InputStream is = s3Client.getObject(request)
    while (true) {
        if (is.read() == -1) {
            break
        }
    }
} catch (Exception e) {
    e.printStackTrace();
}

Using AWS Python SDK (Boto3)

Use AWS Python SDK (Boto3) to access the ObjectNode deployed locally.

Install Boto3 by PIP:

$ pip install boto3

Example: uploading file to ChubaoFS volume (PutObject)

import boto3

endpoint = " ** endpoint url ** "  # example: http://127.0.0.1
region_name = " ** region name ** "
access_key = " ** your access key ** "  # your access key
secret_key = " ** your secret key ** "  # your secret key
bucket = " ** your bucket (volume) ** "  # your volume name
key = " ** your object key (file path in CFS) ** "  # your object name

def put_file():
    session = boto3.Session(
        aws_access_key_id=access_key,
        aws_secret_access_key=secret_key)
    client = session.client(service_name="s3", region_name=region_name, endpoint_url=endpoint)
    client.put_object(Bucket=bucket, Key=key, Body=bytes("hello world"))

Example: downloading file stored in ChubaoFS volume (GetObject)

import boto3

endpoint = " ** endpoint url ** "  # example: http://127.0.0.1
region_name = " ** region name ** "
access_key = " ** your access key ** "  # your access key
secret_key = " ** your secret key ** "  # your secret key
bucket = " ** your bucket (volume) ** "  # your volume name
key = " ** your object key (file path in CFS) ** "  # your object name

def get_file():
    session = boto3.Session(
        aws_access_key_id=access_key,
        aws_secret_access_key=secret_key)
    client = session.client(service_name="s3", region_name=region_name, endpoint_url=endpoint)
    result = client.get_object(Bucket=bucket, Key=key)
    print(result["Body"].read())

Console

How To Start Console

Start a Console process by execute the server binary of ChubaoFS you built with -c argument and specify configuration file.

nohup cfs-server -c console.json &

Configurations

Properties
Key Type Description Mandatory
role string Role of process and must be set to console Yes
logDir string Path for log file storage Yes
logLevel string Level operation for logging. Default is error No
listen string Port of TCP network to be listen, default is 80 Yes
masterAddr string slice Addresses of master server Yes
objectNodeDomain string object domain for sign url for down Yes
monitor_addr string Prometheus the address Yes
dashboard_addr string console menu forward to Grafana Yes
monitor_app string the tag for monitor, it same as master config Yes
monitor_cluster string the tag for monitor, it same as master config Yes

Example:

{
  "role": "console",
  "logDir": "/cfs/log/",
  "logLevel": "debug",
  "listen": "80",
  "masterAddr": [
    "192.168.0.11:17010",
    "192.168.0.12:17010",
    "192.168.0.13:17010"
  ],
  "monitor_addr": "http://192.168.0.102:9090",
  "dashboard_addr": "http://grafana.chubao.io",
  "monitor_app": "cfs",
  "monitor_cluster": "spark"
}

Notice

  • add 127.0.0.1 console.chubao.io grafana.chubao.io in /etc/hosts
  • you can visit it by http://127.0.0.1:80
  • in console default user is root default password is ChubaoFSRoot
  • If you are upgrading your ChubaoFS from v2.0.0 or earlier version, the password may not be compatible, you can use curl -H “Content-Type:application/json” -X POST –data ‘{“id”:”testuser”,”pwd”:”12345”,”type”:2}’ “http://10.196.59.198:17010/user/create” to create new user to use it

Client

Prerequisite

Insert FUSE kernel module and install libfuse.

modprobe fuse
yum install -y fuse

Prepare Config File

fuse.json

{
  "mountPoint": "/cfs/mountpoint",
  "volName": "ltptest",
  "owner": "ltptest",
  "masterAddr": "10.196.59.198:17010,10.196.59.199:17010,10.196.59.200:17010",
  "logDir": "/cfs/client/log",
  "logLevel": "info",
  "profPort": "27510"
}
Supported Configurations
Name Type Description Mandatory
mountPoint string Mount point Yes
volName string Volume name Yes
owner string Owner name as authentication Yes
masterAddr string Resource manager IP address Yes
logDir string Path to store log files No
logLevel string Log level：debug, info, warn, error No
profPort string Golang pprof port No
exporterPort string Performance monitor port No
consulAddr string Performance monitor server address No
lookupValid string Lookup valid duration in FUSE kernel module, unit: sec No
attrValid string Attr valid duration in FUSE kernel module, unit: sec No
icacheTimeout string Inode cache valid duration in client No
enSyncWrite string Enable DirectIO sync write, i.e. make sure data is fsynced in data node No
autoInvalData string Use AutoInvalData FUSE mount option No
rdonly bool Mount as read-only file system No
writecache bool Leverage the write cache feature of kernel FUSE. Requires the kernel FUSE module to support write cache. No
keepcache bool Keep kernel page cache. Requires the writecache option is enabled. No
token string Specify the capability of a client instance. No
readRate int Read Rate Limit. Unlimited by default. No
writeRate int Write Rate Limit. Unlimited by default. No
followerRead bool Enable read from follower. False by default. No
accessKey string Access key of user who owns the volume. No
secretKey string Secret key of user who owns the volume. No
disableDcache bool Disable Dentry Cache. False by default. No
subdir string Mount sub directory. No
fsyncOnClose bool Perform fsync upon file close. True by default. No
maxcpus int The maximum number of available CPU cores. Limit the CPU usage of the client process. No
enableXattr bool Enable xattr support. False by default. No
nearRead bool Enable read from the nearer datanode. True by default, but only take effect when followerRead is enabled. No
enablePosixACL bool Enable posix ACL support. False by default. No

Mount

Use the example fuse.json, and client is mounted on the directory /mnt/fuse. All operations to /mnt/fuse would be performed on the backing distributed file system.

./cfs-client -c fuse.json

Unmount

It is recommended to use standard Linux umount command to terminate the mount.

DataPartitionSelector

dpSelectorName and dpSelectorParm allow custom to specify the desired dataPartition selection strategy to write. If custom does not set any strategy or set a wrong strategy, the default strategy(DefaultRandomSelector) will take effect.

Custom can modify DataPartitionSelector through master API, and do not need to remount client.

curl 'http://masterIP:Port/vol/update?name=volName&authKey=VolKey&dpSelectorName=a&dpSelectorParm=b'

dpSelectorName and dpSelectorParm must be modified at the same time.

Monitor

ChubaoFS use prometheus as metrics collector. It simply config as follow in master, metanode, datanode, client’s config file：

{
    "exporterPort": 9505,
    "consulAddr": "http://consul.prometheus-cfs.local"
}
  • exporterPort：prometheus exporter Port. when set, can export prometheus metrics from URL(http://$hostip:$exporterPort/metrics). If not set, prometheus exporter will unavailable；
  • consulAddr: consul register address, it can work with prometheus to auto discover deployed ChubaoFS nodes, if not set, consul register will not work.

Using grafana as prometheus metrics web front：

_images/cfs-grafana-dashboard.png

Also, user can use prometheus alertmanager to capture and route alerts to the correct receiver. please visit prometheus alertmanger web-doc

Metrics

  • Cluster

    • The number of nodes: MasterCount , MetaNodeCount , DataNodeCount , ObjectNodeCount
    • The number of clients: ClientCount
    • The number of volumes: VolumeCount
    • The size of nodes: DataNodeSize , MetaNodeSize
    • The used ratio of nodes: DataNodeUsedRatio , MetaNodeUsedRatio
    • The number of inactive nodes: DataNodeInactive , MetaNodesInactive
    • The capacity of volumes: VolumeTotalSize
    • The used ratio of volumes: VolumeUsedRatio
    • The number of error disks: DiskError

  • Volume

    • The used size of volumes: VolumeUsedSize
    • The used ratio of volumes: VolumeUsedRatio
    • The capacity change rate of volumes: VolumeSizeRate

  • Master

    • The number of invalid masters: master_nodes_invalid
    • The number of invalid metanodes: metanode_inactive
    • The number of invalid datanodes:： datanode_inactive
    • The number of invalid clients:： fuseclient_inactive

  • MetaNode

    • The duration of each operation (Time) and the number of operations per second (Ops) on the metanode, which can be selected from the MetaNodeOp drop-down list.

  • DataNode

    • The duration of each operation (Time) and the number of operations per second (Ops) on the datanode, which can be selected from the DataNodeOp drop-down list.

  • ObjectNode

    • The duration of each operation (Time) and the number of operations per second (Ops) on the objectnode, which can be selected from the objectNodeOp drop-down list.

  • FuseClient

    • The duration of each operation (Time) and the number of operations per second (Ops) on the client, which can be selected from the fuseOp drop-down list.

Recommended focus metrics: cluster status, node or disk failure, total size, growth rate, etc.

Grafana DashBoard Config

{
  "annotations": {
    "list": [
      {
        "builtIn": 1,
        "datasource": "-- Grafana --",
        "enable": true,
        "hide": true,
        "iconColor": "rgba(0, 211, 255, 1)",
        "name": "Annotations & Alerts",
        "type": "dashboard"
      }
    ]
  },
  "editable": true,
  "gnetId": null,
  "graphTooltip": 0,
  "id": 21,
  "iteration": 1578879879321,
  "links": [],
  "panels": [
    {
      "collapsed": false,
      "gridPos": {
        "h": 1,
        "w": 24,
        "x": 0,
        "y": 0
      },
      "id": 40,
      "panels": [],
      "title": "Cluster",
      "type": "row"
    },
    {
      "cacheTimeout": null,
      "colorBackground": false,
      "colorValue": false,
      "colors": [
        "#299c46",
        "rgba(237, 129, 40, 0.89)",
        "#d44a3a"
      ],
      "datasource": null,
      "description": "master node total count of cluster",
      "format": "none",
      "gauge": {
        "maxValue": 100,
        "minValue": 0,
        "show": false,
        "thresholdLabels": false,
        "thresholdMarkers": true
      },
      "gridPos": {
        "h": 4,
        "w": 4,
        "x": 0,
        "y": 1
      },
      "id": 38,
      "interval": null,
      "links": [],
      "mappingType": 1,
      "mappingTypes": [
        {
          "name": "value to text",
          "value": 1
        },
        {
          "name": "range to text",
          "value": 2
        }
      ],
      "maxDataPoints": 100,
      "nullPointMode": "connected",
      "nullText": null,
      "postfix": "",
      "postfixFontSize": "50%",
      "prefix": "",
      "prefixFontSize": "50%",
      "rangeMaps": [
        {
          "from": "null",
          "text": "N/A",
          "to": "null"
        }
      ],
      "sparkline": {
        "fillColor": "rgba(31, 118, 189, 0.18)",
        "full": false,
        "lineColor": "rgb(31, 120, 193)",
        "show": true
      },
      "tableColumn": "",
      "targets": [
        {
          "expr": "count(go_info{cluster=~\"$cluster\", app=~\"$app\", role=~\"master\"})",
          "format": "time_series",
          "intervalFactor": 1,
          "refId": "A"
        }
      ],
      "thresholds": "",
      "title": "MasterCount",
      "type": "singlestat",
      "valueFontSize": "80%",
      "valueMaps": [
        {
          "op": "=",
          "text": "N/A",
          "value": "null"
        }
      ],
      "valueName": "current"
    },
    {
      "cacheTimeout": null,
      "colorBackground": false,
      "colorValue": false,
      "colors": [
        "#299c46",
        "rgba(237, 129, 40, 0.89)",
        "#d44a3a"
      ],
      "datasource": null,
      "format": "none",
      "gauge": {
        "maxValue": 100,
        "minValue": 0,
        "show": false,
        "thresholdLabels": false,
        "thresholdMarkers": true
      },
      "gridPos": {
        "h": 4,
        "w": 4,
        "x": 4,
        "y": 1
      },
      "id": 42,
      "interval": null,
      "links": [],
      "mappingType": 1,
      "mappingTypes": [
        {
          "name": "value to text",
          "value": 1
        },
        {
          "name": "range to text",
          "value": 2
        }
      ],
      "maxDataPoints": 100,
      "nullPointMode": "connected",
      "nullText": null,
      "postfix": "",
      "postfixFontSize": "50%",
      "prefix": "",
      "prefixFontSize": "50%",
      "rangeMaps": [
        {
          "from": "null",
          "text": "N/A",
          "to": "null"
        }
      ],
      "sparkline": {
        "fillColor": "rgba(31, 118, 189, 0.18)",
        "full": false,
        "lineColor": "rgb(31, 120, 193)",
        "show": true
      },
      "tableColumn": "",
      "targets": [
        {
          "expr": "count(go_info{cluster=~\"$cluster\", app=~\"$app\", role=~\"metanode\"})",
          "format": "time_series",
          "intervalFactor": 1,
          "refId": "A"
        }
      ],
      "thresholds": "",
      "title": "MetanodeCount",
      "type": "singlestat",
      "valueFontSize": "80%",
      "valueMaps": [
        {
          "op": "=",
          "text": "N/A",
          "value": "null"
        }
      ],
      "valueName": "current"
    },
    {
      "cacheTimeout": null,
      "colorBackground": false,
      "colorValue": false,
      "colors": [
        "#299c46",
        "rgba(237, 129, 40, 0.89)",
        "#d44a3a"
      ],
      "datasource": "default",
      "format": "none",
      "gauge": {
        "maxValue": 100,
        "minValue": 0,
        "show": false,
        "thresholdLabels": false,
        "thresholdMarkers": true
      },
      "gridPos": {
        "h": 4,
        "w": 4,
        "x": 8,
        "y": 1
      },
      "id": 41,
      "interval": null,
      "links": [],
      "mappingType": 1,
      "mappingTypes": [
        {
          "name": "value to text",
          "value": 1
        },
        {
          "name": "range to text",
          "value": 2
        }
      ],
      "maxDataPoints": 100,
      "nullPointMode": "connected",
      "nullText": null,
      "postfix": "",
      "postfixFontSize": "50%",
      "prefix": "",
      "prefixFontSize": "50%",
      "rangeMaps": [
        {
          "from": "null",
          "text": "N/A",
          "to": "null"
        }
      ],
      "sparkline": {
        "fillColor": "rgba(31, 118, 189, 0.18)",
        "full": false,
        "lineColor": "rgb(31, 120, 193)",
        "show": true
      },
      "tableColumn": "",
      "targets": [
        {
          "expr": "count(go_info{cluster=\"$cluster\", app=\"$app\", role=\"dataNode\"})",
          "format": "time_series",
          "intervalFactor": 1,
          "refId": "A"
        }
      ],
      "thresholds": "",
      "title": "DatanodeCount",
      "type": "singlestat",
      "valueFontSize": "80%",
      "valueMaps": [
        {
          "op": "=",
          "text": "N/A",
          "value": "null"
        }
      ],
      "valueName": "current"
    },
    {
      "cacheTimeout": null,
      "colorBackground": false,
      "colorValue": false,
      "colors": [
        "#299c46",
        "rgba(237, 129, 40, 0.89)",
        "#d44a3a"
      ],
      "datasource": "default",
      "format": "none",
      "gauge": {
        "maxValue": 100,
        "minValue": 0,
        "show": false,
        "thresholdLabels": false,
        "thresholdMarkers": true
      },
      "gridPos": {
        "h": 4,
        "w": 3,
        "x": 12,
        "y": 1
      },
      "id": 86,
      "interval": null,
      "links": [],
      "mappingType": 1,
      "mappingTypes": [
        {
          "name": "value to text",
          "value": 1
        },
        {
          "name": "range to text",
          "value": 2
        }
      ],
      "maxDataPoints": 100,
      "nullPointMode": "connected",
      "nullText": null,
      "postfix": "",
      "postfixFontSize": "50%",
      "prefix": "",
      "prefixFontSize": "50%",
      "rangeMaps": [
        {
          "from": "null",
          "text": "N/A",
          "to": "null"
        }
      ],
      "sparkline": {
        "fillColor": "rgba(31, 118, 189, 0.18)",
        "full": false,
        "lineColor": "rgb(31, 120, 193)",
        "show": true
      },
      "tableColumn": "",
      "targets": [
        {
          "expr": "count(go_info{cluster=~\"$cluster\", app=~\"$app\", role=~\"fuseclient\"})",
          "format": "time_series",
          "intervalFactor": 1,
          "refId": "A"
        }
      ],
      "thresholds": "",
      "title": "ClientCount",
      "type": "singlestat",
      "valueFontSize": "80%",
      "valueMaps": [
        {
          "op": "=",
          "text": "N/A",
          "value": "null"
        }
      ],
      "valueName": "current"
    },
    {
      "cacheTimeout": null,
      "colorBackground": false,
      "colorValue": false,
      "colors": [
        "#299c46",
        "rgba(237, 129, 40, 0.89)",
        "#d44a3a"
      ],
      "datasource": "default",
      "description": "Cluster volume total count",
      "format": "none",
      "gauge": {
        "maxValue": 100,
        "minValue": 0,
        "show": false,
        "thresholdLabels": false,
        "thresholdMarkers": true
      },
      "gridPos": {
        "h": 4,
        "w": 3,
        "x": 15,
        "y": 1
      },
      "id": 93,
      "interval": null,
      "links": [],
      "mappingType": 1,
      "mappingTypes": [
        {
          "name": "value to text",
          "value": 1
        },
        {
          "name": "range to text",
          "value": 2
        }
      ],
      "maxDataPoints": 100,
      "nullPointMode": "connected",
      "nullText": null,
      "postfix": "",
      "postfixFontSize": "50%",
      "prefix": "",
      "prefixFontSize": "50%",
      "rangeMaps": [
        {
          "from": "null",
          "text": "N/A",
          "to": "null"
        }
      ],
      "sparkline": {
        "fillColor": "rgba(31, 118, 189, 0.18)",
        "full": false,
        "lineColor": "rgb(31, 120, 193)",
        "show": true
      },
      "tableColumn": "",
      "targets": [
        {
          "expr": "sum([[app]]_master_vol_count{cluster=~\"$cluster\"})",
          "format": "time_series",
          "intervalFactor": 1,
          "refId": "A"
        }
      ],
      "thresholds": "",
      "title": "VolumeCount",
      "type": "singlestat",
      "valueFontSize": "80%",
      "valueMaps": [
        {
          "op": "=",
          "text": "N/A",
          "value": "null"
        }
      ],
      "valueName": "current"
    },
    {
      "cacheTimeout": null,
      "colorBackground": false,
      "colorValue": false,
      "colors": [
        "#299c46",
        "rgba(237, 129, 40, 0.89)",
        "#d44a3a"
      ],
      "datasource": null,
      "decimals": null,
      "format": "dateTimeFromNow",
      "gauge": {
        "maxValue": 100,
        "minValue": 0,
        "show": false,
        "thresholdLabels": false,
        "thresholdMarkers": true
      },
      "gridPos": {
        "h": 4,
        "w": 3,
        "x": 18,
        "y": 1
      },
      "id": 113,
      "interval": null,
      "links": [],
      "mappingType": 1,
      "mappingTypes": [
        {
          "name": "value to text",
          "value": 1
        },
        {
          "name": "range to text",
          "value": 2
        }
      ],
      "maxDataPoints": 100,
      "nullPointMode": "connected",
      "nullText": null,
      "postfix": "",
      "postfixFontSize": "50%",
      "prefix": "",
      "prefixFontSize": "50%",
      "rangeMaps": [
        {
          "from": "null",
          "text": "N/A",
          "to": "null"
        }
      ],
      "sparkline": {
        "fillColor": "rgba(31, 118, 189, 0.18)",
        "full": false,
        "lineColor": "rgb(31, 120, 193)",
        "show": true
      },
      "tableColumn": "",
      "targets": [
        {
          "expr": "process_start_time_seconds{cluster=\"$cluster\", app=\"$app\", instance=~\"$instance\"}*1000",
          "format": "time_series",
          "intervalFactor": 1,
          "refId": "A"
        }
      ],
      "thresholds": "",
      "title": "ProcessStartTime",
      "type": "singlestat",
      "valueFontSize": "80%",
      "valueMaps": [
        {
          "op": "=",
          "text": "N/A",
          "value": "null"
        }
      ],
      "valueName": "current"
    },
    {
      "aliasColors": {},
      "bars": false,
      "dashLength": 10,
      "dashes": false,
      "datasource": null,
      "fill": 1,
      "gridPos": {
        "h": 6,
        "w": 7,
        "x": 0,
        "y": 5
      },
      "id": 115,
      "legend": {
        "avg": false,
        "current": false,
        "max": false,
        "min": false,
        "show": true,
        "total": false,
        "values": false
      },
      "lines": true,
      "linewidth": 1,
      "links": [],
      "nullPointMode": "null",
      "percentage": false,
      "pointradius": 5,
      "points": false,
      "renderer": "flot",
      "seriesOverrides": [],
      "spaceLength": 10,
      "stack": false,
      "steppedLine": false,
      "targets": [
        {
          "expr": "sum(go_info{app=\"$app\", cluster=\"$cluster\"}) by (cluster, app, role)",
          "format": "time_series",
          "intervalFactor": 1,
          "legendFormat": "{{cluster}}-{{role}}",
          "refId": "A"
        },
        {
          "expr": "count(go_info{cluster=~\"$cluster\", app=~\"$app\", role=~\"$role\"})",
          "format": "time_series",
          "hide": true,
          "intervalFactor": 1,
          "legendFormat": "[[role]]-count",
          "refId": "B"
        }
      ],
      "thresholds": [],
      "timeFrom": null,
      "timeShift": null,
      "title": "NodeAlive",
      "tooltip": {
        "shared": true,
        "sort": 0,
        "value_type": "individual"
      },
      "type": "graph",
      "xaxis": {
        "buckets": null,
        "mode": "time",
        "name": null,
        "show": true,
        "values": []
      },
      "yaxes": [
        {
          "format": "locale",
          "label": null,
          "logBase": 1,
          "max": null,
          "min": null,
          "show": true
        },
        {
          "format": "short",
          "label": null,
          "logBase": 1,
          "max": null,
          "min": null,
          "show": true
        }
      ],
      "yaxis": {
        "align": false,
        "alignLevel": null
      }
    },
    {
      "aliasColors": {},
      "bars": false,
      "dashLength": 10,
      "dashes": false,
      "datasource": "default",
      "fill": 1,
      "gridPos": {
        "h": 6,
        "w": 7,
        "x": 7,
        "y": 5
      },
      "id": 179,
      "legend": {
        "avg": false,
        "current": false,
        "max": false,
        "min": false,
        "show": true,
        "total": false,
        "values": false
      },
      "lines": true,
      "linewidth": 1,
      "links": [],
      "nullPointMode": "null",
      "percentage": false,
      "pointradius": 5,
      "points": false,
      "renderer": "flot",
      "seriesOverrides": [
        {
          "alias": "sum"
        }
      ],
      "spaceLength": 10,
      "stack": false,
      "steppedLine": false,
      "targets": [
        {
          "expr": "sum(cfs_master_disk_error{app=\"$app\", cluster=\"$cluster\"} > 0)",
          "format": "time_series",
          "intervalFactor": 1,
          "legendFormat": "sum",
          "refId": "D"
        },
        {
          "expr": "cfs_master_disk_error{app=\"$app\", cluster=\"$cluster\"} > 0",
          "format": "time_series",
          "intervalFactor": 1,
          "legendFormat": "{{addr}}_{{path}}",
          "refId": "A"
        }
      ],
      "thresholds": [],
      "timeFrom": null,
      "timeShift": null,
      "title": "DiskError",
      "tooltip": {
        "shared": true,
        "sort": 0,
        "value_type": "individual"
      },
      "type": "graph",
      "xaxis": {
        "buckets": null,
        "mode": "time",
        "name": null,
        "show": true,
        "values": []
      },
      "yaxes": [
        {
          "format": "locale",
          "label": null,
          "logBase": 1,
          "max": null,
          "min": null,
          "show": true
        },
        {
          "format": "locale",
          "label": null,
          "logBase": 1,
          "max": null,
          "min": null,
          "show": true
        }
      ],
      "yaxis": {
        "align": false,
        "alignLevel": null
      }
    },
    {
      "aliasColors": {},
      "bars": false,
      "dashLength": 10,
      "dashes": false,
      "datasource": "default",
      "fill": 1,
      "gridPos": {
        "h": 6,
        "w": 7,
        "x": 14,
        "y": 5
      },
      "id": 180,
      "legend": {
        "avg": false,
        "current": false,
        "max": false,
        "min": false,
        "show": true,
        "total": false,
        "values": false
      },
      "lines": true,
      "linewidth": 1,
      "links": [],
      "nullPointMode": "null",
      "percentage": false,
      "pointradius": 5,
      "points": false,
      "renderer": "flot",
      "seriesOverrides": [],
      "spaceLength": 10,
      "stack": false,
      "steppedLine": false,
      "targets": [
        {
          "expr": "cfs_master_metaNodes_inactive{app=\"$app\", cluster=\"$cluster\"}> 0",
          "format": "time_series",
          "hide": false,
          "intervalFactor": 1,
          "legendFormat": "{{instance}}",
          "refId": "B"
        },
        {
          "expr": "sum(cfs_master_metaNodes_inactive{app=\"$app\", cluster=\"$cluster\"}> 0)",
          "format": "time_series",
          "hide": false,
          "intervalFactor": 1,
          "legendFormat": "sum",
          "refId": "A"
        }
      ],
      "thresholds": [],
      "timeFrom": null,
      "timeShift": null,
      "title": "MetaNodesInactive",
      "tooltip": {
        "shared": true,
        "sort": 0,
        "value_type": "individual"
      },
      "type": "graph",
      "xaxis": {
        "buckets": null,
        "mode": "time",
        "name": null,
        "show": true,
        "values": []
      },
      "yaxes": [
        {
          "format": "locale",
          "label": null,
          "logBase": 1,
          "max": null,
          "min": null,
          "show": true
        },
        {
          "format": "short",
          "label": null,
          "logBase": 1,
          "max": null,
          "min": null,
          "show": true
        }
      ],
      "yaxis": {
        "align": false,
        "alignLevel": null
      }
    },
    {
      "aliasColors": {},
      "bars": false,
      "dashLength": 10,
      "dashes": false,
      "datasource": "default",
      "fill": 1,
      "gridPos": {
        "h": 6,
        "w": 7,
        "x": 0,
        "y": 11
      },
      "id": 181,
      "legend": {
        "avg": false,
        "current": false,
        "max": false,
        "min": false,
        "show": true,
        "total": false,
        "values": false
      },
      "lines": true,
      "linewidth": 1,
      "links": [],
      "nullPointMode": "null",
      "percentage": false,
      "pointradius": 5,
      "points": false,
      "renderer": "flot",
      "seriesOverrides": [],
      "spaceLength": 10,
      "stack": false,
      "steppedLine": false,
      "targets": [
        {
          "expr": "cfs_master_dataNodes_inactive{app=\"$app\", cluster=\"$cluster\"}> 0",
          "format": "time_series",
          "hide": false,
          "intervalFactor": 1,
          "legendFormat": "{{instance}}",
          "refId": "C"
        },
        {
          "expr": "sum(cfs_master_dataNodes_inactive{app=\"$app\", cluster=\"$cluster\"}> 0)",
          "format": "time_series",
          "hide": false,
          "intervalFactor": 1,
          "legendFormat": "sum",
          "refId": "A"
        }
      ],
      "thresholds": [],
      "timeFrom": null,
      "timeShift": null,
      "title": "DatanodeInactive",
      "tooltip": {
        "shared": true,
        "sort": 0,
        "value_type": "individual"
      },
      "type": "graph",
      "xaxis": {
        "buckets": null,
        "mode": "time",
        "name": null,
        "show": true,
        "values": []
      },
      "yaxes": [
        {
          "format": "locale",
          "label": null,
          "logBase": 1,
          "max": null,
          "min": null,
          "show": true
        },
        {
          "format": "short",
          "label": null,
          "logBase": 1,
          "max": null,
          "min": null,
          "show": true
        }
      ],
      "yaxis": {
        "align": false,
        "alignLevel": null
      }
    },
    {
      "aliasColors": {},
      "bars": false,
      "dashLength": 10,
      "dashes": false,
      "datasource": null,
      "fill": 1,
      "gridPos": {
        "h": 6,
        "w": 7,
        "x": 7,
        "y": 11
      },
      "id": 177,
      "legend": {
        "avg": false,
        "current": false,
        "max": false,
        "min": false,
        "show": true,
        "total": false,
        "values": false
      },
      "lines": true,
      "linewidth": 1,
      "links": [],
      "nullPointMode": "null",
      "percentage": false,
      "pointradius": 5,
      "points": false,
      "renderer": "flot",
      "seriesOverrides": [
        {
          "alias": "/used_ratio.*/",
          "yaxis": 2
        }
      ],
      "spaceLength": 10,
      "stack": false,
      "steppedLine": false,
      "targets": [
        {
          "expr": "cfs_master_dataNodes_total_GB{app=\"$app\",cluster=\"$cluster\"}>0",
          "format": "time_series",
          "hide": false,
          "intervalFactor": 1,
          "legendFormat": "TotalGB",
          "refId": "B"
        },
        {
          "expr": "cfs_master_dataNodes_used_GB{app=\"$app\",cluster=\"$cluster\"}>0",
          "format": "time_series",
          "hide": false,
          "intervalFactor": 1,
          "legendFormat": "UsedGB",
          "refId": "A"
        },
        {
          "expr": "cfs_master_dataNodes_increased_GB{app=\"$app\",cluster=\"$cluster\"}>0",
          "format": "time_series",
          "hide": false,
          "intervalFactor": 1,
          "legendFormat": "IncreasedGB",
          "refId": "C"
        },
        {
          "expr": "sum(cfs_master_dataNodes_used_GB{app=\"$app\",cluster=\"$cluster\"}) / sum(cfs_master_dataNodes_total_GB{app=\"$app\",cluster=\"$cluster\"})",
          "format": "time_series",
          "hide": false,
          "intervalFactor": 1,
          "legendFormat": "UsedRatio",
          "refId": "D"
        }
      ],
      "thresholds": [],
      "timeFrom": null,
      "timeShift": null,
      "title": "DatanodeSize",
      "tooltip": {
        "shared": true,
        "sort": 0,
        "value_type": "individual"
      },
      "type": "graph",
      "xaxis": {
        "buckets": null,
        "mode": "time",
        "name": null,
        "show": true,
        "values": []
      },
      "yaxes": [
        {
          "format": "decgbytes",
          "label": null,
          "logBase": 1,
          "max": null,
          "min": null,
          "show": true
        },
        {
          "format": "percentunit",
          "label": null,
          "logBase": 1,
          "max": null,
          "min": null,
          "show": true
        }
      ],
      "yaxis": {
        "align": false,
        "alignLevel": null
      }
    },
    {
      "aliasColors": {},
      "bars": false,
      "dashLength": 10,
      "dashes": false,
      "datasource": null,
      "fill": 1,
      "gridPos": {
        "h": 6,
        "w": 7,
        "x": 14,
        "y": 11
      },
      "id": 92,
      "legend": {
        "avg": false,
        "current": false,
        "max": false,
        "min": false,
        "show": true,
        "total": false,
        "values": false
      },
      "lines": true,
      "linewidth": 1,
      "links": [],
      "nullPointMode": "null",
      "percentage": false,
      "pointradius": 5,
      "points": false,
      "renderer": "flot",
      "seriesOverrides": [],
      "spaceLength": 10,
      "stack": false,
      "steppedLine": false,
      "targets": [
        {
          "expr": "sum([[app]]_master_metaNodes_total_GB{cluster=~\"$cluster\"})",
          "format": "time_series",
          "intervalFactor": 1,
          "legendFormat": "total_GB",
          "refId": "B"
        },
        {
          "expr": "sum([[app]]_master_metaNodes_used_GB{cluster=~\"$cluster\"})",
          "format": "time_series",
          "intervalFactor": 1,
          "legendFormat": "used_GB",
          "refId": "A"
        },
        {
          "expr": "sum([[app]]_master_metaNodes_increased_GB{cluster=~\"$cluster\"})",
          "format": "time_series",
          "intervalFactor": 1,
          "legendFormat": "increased_GB",
          "refId": "C"
        }
      ],
      "thresholds": [],
      "timeFrom": null,
      "timeShift": null,
      "title": "MetanodeSize",
      "tooltip": {
        "shared": true,
        "sort": 0,
        "value_type": "individual"
      },
      "type": "graph",
      "xaxis": {
        "buckets": null,
        "mode": "time",
        "name": null,
        "show": true,
        "values": []
      },
      "yaxes": [
        {
          "format": "decgbytes",
          "label": null,
          "logBase": 1,
          "max": null,
          "min": null,
          "show": true
        },
        {
          "format": "short",
          "label": null,
          "logBase": 1,
          "max": null,
          "min": null,
          "show": true
        }
      ],
      "yaxis": {
        "align": false,
        "alignLevel": null
      }
    },
    {
      "aliasColors": {},
      "bars": false,
      "dashLength": 10,
      "dashes": false,
      "datasource": null,
      "fill": 1,
      "gridPos": {
        "h": 6,
        "w": 7,
        "x": 0,
        "y": 17
      },
      "id": 175,
      "legend": {
        "avg": false,
        "current": false,
        "max": false,
        "min": false,
        "show": true,
        "total": false,
        "values": false
      },
      "lines": true,
      "linewidth": 1,
      "links": [],
      "nullPointMode": "null",
      "percentage": false,
      "pointradius": 5,
      "points": false,
      "renderer": "flot",
      "seriesOverrides": [
        {
          "alias": "/.*_ratio.*/",
          "yaxis": 2
        }
      ],
      "spaceLength": 10,
      "stack": false,
      "steppedLine": false,
      "targets": [
        {
          "expr": "cfs_master_vol_usage_ratio{app=\"$app\",cluster=\"$cluster\"}",
          "format": "time_series",
          "hide": true,
          "intervalFactor": 1,
          "legendFormat": "{{volName}}_usage_ratio",
          "refId": "B"
        },
        {
          "expr": "cfs_master_vol_total_GB{app=\"$app\",cluster=\"$cluster\"}",
          "format": "time_series",
          "hide": false,
          "intervalFactor": 1,
          "legendFormat": "{{volName}}_total_GB",
          "refId": "A"
        },
        {
          "expr": "cfs_master_vol_used_GB{app=\"$app\",cluster=\"$cluster\"}",
          "format": "time_series",
          "hide": false,
          "intervalFactor": 1,
          "legendFormat": "{{volName}}_used_GB",
          "refId": "C"
        },
        {
          "expr": "cfs_master_vol_used_GB{app=\"$app\",cluster=\"$cluster\"} / cfs_master_vol_total_GB{app=\"$app\",cluster=\"$cluster\"} ",
          "format": "time_series",
          "hide": false,
          "intervalFactor": 1,
          "legendFormat": "{{volName}}_used_ratio",
          "refId": "D"
        }
      ],
      "thresholds": [],
      "timeFrom": null,
      "timeShift": null,
      "title": "VolumeSize",
      "tooltip": {
        "shared": true,
        "sort": 0,
        "value_type": "individual"
      },
      "type": "graph",
      "xaxis": {
        "buckets": null,
        "mode": "time",
        "name": null,
        "show": true,
        "values": []
      },
      "yaxes": [
        {
          "format": "decgbytes",
          "label": null,
          "logBase": 1,
          "max": null,
          "min": null,
          "show": true
        },
        {
          "format": "percentunit",
          "label": null,
          "logBase": 1,
          "max": null,
          "min": null,
          "show": true
        }
      ],
      "yaxis": {
        "align": false,
        "alignLevel": null
      }
    },
    {
      "aliasColors": {},
      "bars": false,
      "dashLength": 10,
      "dashes": false,
      "datasource": "default",
      "fill": 1,
      "gridPos": {
        "h": 6,
        "w": 7,
        "x": 7,
        "y": 17
      },
      "id": 90,
      "legend": {
        "avg": false,
        "current": false,
        "max": false,
        "min": false,
        "show": true,
        "total": false,
        "values": false
      },
      "lines": true,
      "linewidth": 1,
      "links": [],
      "nullPointMode": "null",
      "percentage": false,
      "pointradius": 5,
      "points": false,
      "renderer": "flot",
      "seriesOverrides": [],
      "spaceLength": 10,
      "stack": false,
      "steppedLine": false,
      "targets": [
        {
          "expr": "sum([[app]]_master_vol_count{cluster=~\"$cluster\"})",
          "format": "time_series",
          "intervalFactor": 1,
          "legendFormat": "vol_count",
          "refId": "M"
        }
      ],
      "thresholds": [],
      "timeFrom": null,
      "timeShift": null,
      "title": "VolumeCount",
      "tooltip": {
        "shared": true,
        "sort": 0,
        "value_type": "individual"
      },
      "type": "graph",
      "xaxis": {
        "buckets": null,
        "mode": "time",
        "name": null,
        "show": true,
        "values": []
      },
      "yaxes": [
        {
          "format": "locale",
          "label": null,
          "logBase": 1,
          "max": null,
          "min": null,
          "show": true
        },
        {
          "format": "short",
          "label": null,
          "logBase": 1,
          "max": null,
          "min": null,
          "show": true
        }
      ],
      "yaxis": {
        "align": false,
        "alignLevel": null
      }
    },
    {
      "aliasColors": {},
      "bars": false,
      "dashLength": 10,
      "dashes": false,
      "datasource": "default",
      "fill": 1,
      "gridPos": {
        "h": 6,
        "w": 7,
        "x": 14,
        "y": 17
      },
      "id": 73,
      "legend": {
        "avg": false,
        "current": false,
        "max": false,
        "min": false,
        "show": true,
        "total": false,
        "values": false
      },
      "lines": true,
      "linewidth": 1,
      "links": [],
      "nullPointMode": "null",
      "percentage": false,
      "pointradius": 5,
      "points": false,
      "renderer": "flot",
      "seriesOverrides": [
        {
          "alias": "/.*_QPS/",
          "yaxis": 2
        }
      ],
      "spaceLength": 10,
      "stack": false,
      "steppedLine": false,
      "targets": [
        {
          "expr": "sum([[app]]_metanode_OpMetaOpen{cluster=~\"$cluster\"})",
          "format": "time_series",
          "intervalFactor": 1,
          "legendFormat": "Open",
          "refId": "A"
        },
        {
          "expr": "sum([[app]]_metanode_OpMetaLookup{cluster=~\"$cluster\"})",
          "format": "time_series",
          "intervalFactor": 1,
          "legendFormat": "Lookup",
          "refId": "B"
        },
        {
          "expr": "sum([[app]]_metanode_OpMetaNodeHeartbeat{cluster=~\"$cluster\"})",
          "format": "time_series",
          "intervalFactor": 1,
          "legendFormat": "NodeHeartbeat",
          "refId": "C"
        },
        {
          "expr": "sum([[app]]_metanode_OpMetaReadDir{cluster=~\"$cluster\"})",
          "format": "time_series",
          "intervalFactor": 1,
          "legendFormat": "ReadDir",
          "refId": "D"
        },
        {
          "expr": "sum([[app]]_metanode_OpMetaReleaseOpen{cluster=~\"$cluster\"})",
          "format": "time_series",
          "intervalFactor": 1,
          "legendFormat": "ReleaseOpen",
          "refId": "E"
        },
        {
          "expr": "sum([[app]]_metanode_OpMetaSetattr{cluster=~\"$cluster\"})",
          "format": "time_series",
          "intervalFactor": 1,
          "legendFormat": "Setattr",
          "refId": "F"
        },
        {
          "expr": "sum([[app]]_metanode_OpMetaTruncate{cluster=~\"$cluster\"})",
          "format": "time_series",
          "intervalFactor": 1,
          "legendFormat": "Truncate",
          "refId": "G"
        },
        {
          "expr": "sum(rate([[app]]_metanode_OpMetaOpen{cluster=~\"$cluster\"}[1m]))",
          "format": "time_series",
          "intervalFactor": 1,
          "legendFormat": "Open_QPS",
          "refId": "H"
        },
        {
          "expr": "sum(rate([[app]]_metanode_OpMetaLookup{cluster=~\"$cluster\"}[1m]))",
          "format": "time_series",
          "intervalFactor": 1,
          "legendFormat": "Lookup_QPS",
          "refId": "I"
        },
        {
          "expr": "sum(rate([[app]]_metanode_OpMetaReadDir{cluster=~\"$cluster\"}[1m]))",
          "format": "time_series",
          "intervalFactor": 1,
          "legendFormat": "ReadDir_QPS",
          "refId": "K"
        },
        {
          "expr": "sum(rate([[app]]_metanode_OpMetaTruncate{cluster=~\"$cluster\"}[1m]))",
          "format": "time_series",
          "intervalFactor": 1,
          "legendFormat": "Truncate_QPS",
          "refId": "L"
        },
        {
          "expr": "sum(rate([[app]]_metanode_OpMetaReleaseOpen{cluster=~\"$cluster\"}[1m]))",
          "format": "time_series",
          "intervalFactor": 1,
          "legendFormat": "ReleaseOpen_QPS",
          "refId": "J"
        },
        {
          "expr": "sum(rate([[app]]cfs_metanode_OpMetaSetattr{cluster=~\"$cluster\"}[1m]))",
          "format": "time_series",
          "intervalFactor": 1,
          "legendFormat": "Setattr_QPS",
          "refId": "M"
        }
      ],
      "thresholds": [],
      "timeFrom": null,
      "timeShift": null,
      "title": "MetanodeOperations",
      "tooltip": {
        "shared": true,
        "sort": 0,
        "value_type": "individual"
      },
      "type": "graph",
      "xaxis": {
        "buckets": null,
        "mode": "time",
        "name": null,
        "show": true,
        "values": []
      },
      "yaxes": [
        {
          "format": "ns",
          "label": null,
          "logBase": 1,
          "max": null,
          "min": null,
          "show": true
        },
        {
          "format": "ops",
          "label": null,
          "logBase": 1,
          "max": null,
          "min": null,
          "show": true
        }
      ],
      "yaxis": {
        "align": false,
        "alignLevel": null
      }
    },
    {
      "aliasColors": {},
      "bars": false,
      "dashLength": 10,
      "dashes": false,
      "datasource": "default",
      "fill": 1,
      "gridPos": {
        "h": 6,
        "w": 7,
        "x": 0,
        "y": 23
      },
      "id": 71,
      "legend": {
        "avg": false,
        "current": false,
        "max": false,
        "min": false,
        "show": true,
        "total": false,
        "values": false
      },
      "lines": true,
      "linewidth": 1,
      "links": [],
      "nullPointMode": "null",
      "percentage": false,
      "pointradius": 5,
      "points": false,
      "renderer": "flot",
      "seriesOverrides": [
        {
          "alias": "/.*_QPS/",
          "yaxis": 2
        }
      ],
      "spaceLength": 10,
      "stack": false,
      "steppedLine": false,
      "targets": [
        {
          "expr": "sum([[app]]_metanode_OpMetaBatchInodeGet{cluster=~\"$cluster\", instance=\"$instance\"})",
          "format": "time_series",
          "intervalFactor": 1,
          "legendFormat": "BatchGet",
          "refId": "A"
        },
        {
          "expr": "sum([[app]]_metanode_OpMetaCreateInode{cluster=~\"$cluster\", instance=\"$instance\"})",
          "format": "time_series",
          "intervalFactor": 1,
          "legendFormat": "Create",
          "refId": "B"
        },
        {
          "expr": "sum([[app]]_metanode_OpMetaDeleteInode{cluster=~\"$cluster\"})",
          "format": "time_series",
          "intervalFactor": 1,
          "legendFormat": "Delete",
          "refId": "C"
        },
        {
          "expr": "sum([[app]]_metanode_OpMetaEvictInode{cluster=~\"$cluster\"})",
          "format": "time_series",
          "intervalFactor": 1,
          "legendFormat": "Evict",
          "refId": "D"
        },
        {
          "expr": "sum([[app]]_metanode_OpMetaInodeGet{cluster=~\"$cluster\"})",
          "format": "time_series",
          "intervalFactor": 1,
          "legendFormat": "GetInode",
          "refId": "E"
        },
        {
          "expr": "sum([[app]]_metanode_OpMetaLinkInode{cluster=~\"$cluster\"})",
          "format": "time_series",
          "intervalFactor": 1,
          "legendFormat": "LinkInode",
          "refId": "F"
        },
        {
          "expr": "sum(rate([[app]]_metanode_OpMetaCreateInode{cluster=~\"$cluster\"}[1m]))",
          "format": "time_series",
          "intervalFactor": 1,
          "legendFormat": "CreateInode_QPS",
          "refId": "G"
        },
        {
          "expr": "sum(rate([[app]]_metanode_OpMetaDeleteInode{cluster=~\"$cluster\"}[1m]))",
          "format": "time_series",
          "intervalFactor": 1,
          "legendFormat": "DeleteInode_QPS",
          "refId": "H"
        },
        {
          "expr": "sum(rate([[app]]_metanode_OpMetaEvictInode{cluster=~\"$cluster\"}[1m]))",
          "format": "time_series",
          "intervalFactor": 1,
          "legendFormat": "EvictInode_QPS",
          "refId": "I"
        },
        {
          "expr": "sum(rate([[app]]_metanode_OpMetaInodeGet{cluster=~\"$cluster\"}[1m]))",
          "format": "time_series",
          "intervalFactor": 1,
          "legendFormat": "GetInode_QPS",
          "refId": "J"
        },
        {
          "expr": "sum(rate([[app]]_metanode_OpMetaLinkInode{cluster=~\"$cluster\"}[1m]))",
          "format": "time_series",
          "intervalFactor": 1,
          "legendFormat": "LinkInode_QPS",
          "refId": "K"
        },
        {
          "expr": "sum(rate([[app]]_metanode_OpBatchInodeGet{cluster=~\"$cluster\"}[1m]))",
          "format": "time_series",
          "intervalFactor": 1,
          "legendFormat": "BantchInodeGet_QPS",
          "refId": "L"
        },
        {
          "expr": "sum([[app]]_metanode_OpMetaTruncate{cluster=~\"$cluster\"})",
          "format": "time_series",
          "intervalFactor": 1,
          "legendFormat": "TruncateInode",
          "refId": "M"
        },
        {
          "expr": "sum(rate([[app]]_metanode_OpMetaTruncate{cluster=~\"$cluster\"}[1m]))",
          "format": "time_series",
          "intervalFactor": 1,
          "legendFormat": "TruncateInode_QPS",
          "refId": "N"
        },
        {
          "expr": "sum([[app]]_metanode_OpMetaUnlinkInode{cluster=~\"$cluster\", instance=\"$instance\"})",
          "format": "time_series",
          "intervalFactor": 1,
          "legendFormat": "UnlinkInode",
          "refId": "O"
        },
        {
          "expr": "sum(rate([[app]]_metanode_OpMetaUnlinkInode{cluster=~\"$cluster\"}[1m]))",
          "format": "time_series",
          "intervalFactor": 1,
          "legendFormat": "UnlinkInnode_QPS",
          "refId": "P"
        }
      ],
      "thresholds": [],
      "timeFrom": null,
      "timeShift": null,
      "title": "MetanodeInodeOperations",
      "tooltip": {
        "shared": true,
        "sort": 0,
        "value_type": "individual"
      },
      "type": "graph",
      "xaxis": {
        "buckets": null,
        "mode": "time",
        "name": null,
        "show": true,
        "values": []
      },
      "yaxes": [
        {
          "format": "ns",
          "label": null,
          "logBase": 1,
          "max": null,
          "min": null,
          "show": true
        },
        {
          "format": "ops",
          "label": null,
          "logBase": 1,
          "max": null,
          "min": null,
          "show": true
        }
      ],
      "yaxis": {
        "align": false,
        "alignLevel": null
      }
    },
    {
      "aliasColors": {},
      "bars": false,
      "dashLength": 10,
      "dashes": false,
      "datasource": null,
      "fill": 1,
      "gridPos": {
        "h": 6,
        "w": 7,
        "x": 7,
        "y": 23
      },
      "id": 45,
      "legend": {
        "avg": false,
        "current": false,
        "max": false,
        "min": false,
        "show": true,
        "total": false,
        "values": false
      },
      "lines": true,
      "linewidth": 1,
      "links": [],
      "nullPointMode": "null",
      "percentage": false,
      "pointradius": 5,
      "points": false,
      "renderer": "flot",
      "seriesOverrides": [
        {
          "alias": "/.*_QPS/",
          "yaxis": 2
        }
      ],
      "spaceLength": 10,
      "stack": false,
      "steppedLine": false,
      "targets": [
        {
          "expr": "sum([[app]]_metanode_OpMetaCreateDentry{cluster=~\"$cluster\"})",
          "format": "time_series",
          "intervalFactor": 1,
          "legendFormat": "CreateDentryCreateDentry",
          "refId": "A"
        },
        {
          "expr": "sum([[app]]_metanode_OpMetaDeleteDentry{cluster=~\"$cluster\"})",
          "format": "time_series",
          "intervalFactor": 1,
          "legendFormat": "DeleteDentry",
          "refId": "B"
        },
        {
          "expr": "sum([[app]]_metanode_OpMetaUpdateDentry{cluster=~\"$cluster\"})",
          "format": "time_series",
          "intervalFactor": 1,
          "legendFormat": "UpdateDentry",
          "refId": "C"
        },
        {
          "expr": "sum(rate([[app]]_metanode_OpMetaCreateDentry{cluster=~\"$cluster\"}[1m]))",
          "format": "time_series",
          "intervalFactor": 1,
          "legendFormat": "CreateDentry_QPS",
          "refId": "D"
        },
        {
          "expr": "sum(rate([[app]]_metanode_OpMetaDeleteDentry{cluster=~\"$cluster\"}[1m]))",
          "format": "time_series",
          "intervalFactor": 1,
          "legendFormat": "DeleteDentry_QPS",
          "refId": "E"
        },
        {
          "expr": "sum(rate([[app]]_metanode_OpMetaUpdateDentry{cluster=~\"$cluster\"}[1m]))",
          "format": "time_series",
          "intervalFactor": 1,
          "legendFormat": "UpdateDentry_QPS",
          "refId": "F"
        }
      ],
      "thresholds": [],
      "timeFrom": null,
      "timeShift": null,
      "title": "MetanodeDentryOperation",
      "tooltip": {
        "shared": true,
        "sort": 0,
        "value_type": "individual"
      },
      "type": "graph",
      "xaxis": {
        "buckets": null,
        "mode": "time",
        "name": null,
        "show": true,
        "values": []
      },
      "yaxes": [
        {
          "format": "ns",
          "label": null,
          "logBase": 1,
          "max": null,
          "min": null,
          "show": true
        },
        {
          "format": "ops",
          "label": null,
          "logBase": 1,
          "max": null,
          "min": null,
          "show": true
        }
      ],
      "yaxis": {
        "align": false,
        "alignLevel": null
      }
    },
    {
      "aliasColors": {},
      "bars": false,
      "dashLength": 10,
      "dashes": false,
      "datasource": "default",
      "fill": 1,
      "gridPos": {
        "h": 6,
        "w": 7,
        "x": 14,
        "y": 23
      },
      "id": 79,
      "legend": {
        "avg": false,
        "current": false,
        "max": false,
        "min": false,
        "show": true,
        "total": false,
        "values": false
      },
      "lines": true,
      "linewidth": 1,
      "links": [],
      "nullPointMode": "null",
      "percentage": false,
      "pointradius": 5,
      "points": false,
      "renderer": "flot",
      "seriesOverrides": [
        {
          "alias": "/.*_QPS/",
          "yaxis": 2
        }
      ],
      "spaceLength": 10,
      "stack": false,
      "steppedLine": false,
      "targets": [
        {
          "expr": "sum([[app]]_dataNode_OpCreateExtent{cluster=~\"$cluster\"})",
          "format": "time_series",
          "intervalFactor": 1,
          "legendFormat": "CreateExtent",
          "refId": "A"
        },
        {
          "expr": "sum([[app]]_dataNode_OpMarkDelete{cluster=~\"$cluster\"})",
          "format": "time_series",
          "intervalFactor": 1,
          "legendFormat": "MarkDeleteExtent",
          "refId": "B"
        },
        {
          "expr": "sum([[app]]_dataNode_OpRandomWrite{cluster=~\"$cluster\"})",
          "format": "time_series",
          "intervalFactor": 1,
          "legendFormat": "RandomWrite",
          "refId": "D"
        },
        {
          "expr": "sum([[app]]_dataNode_OpWrite{cluster=~\"$cluster\"})",
          "format": "time_series",
          "intervalFactor": 1,
          "legendFormat": "AppendWrite",
          "refId": "C"
        },
        {
          "expr": "sum([[app]]_dataNode_OpStreamRead{cluster=~\"$cluster\"})",
          "format": "time_series",
          "intervalFactor": 1,
          "legendFormat": "Read",
          "refId": "F"
        },
        {
          "expr": "sum(rate([[app]]_datanode_OpCreateExtent{cluster=~\"$cluster\"}[1m]))",
          "format": "time_series",
          "intervalFactor": 1,
          "legendFormat": "CreateExtent_QPS",
          "refId": "E"
        },
        {
          "expr": "sum(rate([[app]]_dataNode_OpMarkDelete{cluster=~\"$cluster\"}[1m]))",
          "format": "time_series",
          "intervalFactor": 1,
          "legendFormat": "MarkDeleteExtent_QPS",
          "refId": "G"
        },
        {
          "expr": "sum(rate([[app]]_dataNode_OpRandomWrite{cluster=~\"$cluster\"}[1m]))",
          "format": "time_series",
          "intervalFactor": 1,
          "legendFormat": "RandomWrite_QPS",
          "refId": "H"
        },
        {
          "expr": "sum(rate([[app]]_dataNode_OpWrite{cluster=~\"$cluster\"}[1m]))",
          "format": "time_series",
          "intervalFactor": 1,
          "legendFormat": "AppendWrite_QPS",
          "refId": "I"
        },
        {
          "expr": "sum(rate([[app]]_dataNode_OpStreamRead{cluster=~\"$cluster\"}[1m]))",
          "format": "time_series",
          "intervalFactor": 1,
          "legendFormat": "Read_QPS",
          "refId": "J"
        }
      ],
      "thresholds": [],
      "timeFrom": null,
      "timeShift": null,
      "title": "DatanodeExtentOperations",
      "tooltip": {
        "shared": true,
        "sort": 0,
        "value_type": "individual"
      },
      "type": "graph",
      "xaxis": {
        "buckets": null,
        "mode": "time",
        "name": null,
        "show": true,
        "values": []
      },
      "yaxes": [
        {
          "format": "ns",
          "label": null,
          "logBase": 1,
          "max": null,
          "min": null,
          "show": true
        },
        {
          "format": "ops",
          "label": null,
          "logBase": 1,
          "max": null,
          "min": null,
          "show": true
        }
      ],
      "yaxis": {
        "align": false,
        "alignLevel": null
      }
    },
    {
      "aliasColors": {},
      "bars": false,
      "dashLength": 10,
      "dashes": false,
      "datasource": "default",
      "fill": 1,
      "gridPos": {
        "h": 6,
        "w": 7,
        "x": 0,
        "y": 29
      },
      "id": 70,
      "legend": {
        "avg": false,
        "current": false,
        "max": false,
        "min": false,
        "show": true,
        "total": false,
        "values": false
      },
      "lines": true,
      "linewidth": 1,
      "links": [],
      "nullPointMode": "null",
      "percentage": false,
      "pointradius": 5,
      "points": false,
      "renderer": "flot",
      "seriesOverrides": [],
      "spaceLength": 10,
      "stack": false,
      "steppedLine": false,
      "targets": [
        {
          "expr": "sum([[app]]_metanode_OpCreateMetaPartition{cluster=~\"$cluster\", instance=\"$instance\"})",
          "format": "time_series",
          "intervalFactor": 1,
          "legendFormat": "Create",
          "refId": "A"
        },
        {
          "expr": "sum([[app]]_metanode_OpLoadMetaPartition{cluster=~\"$cluster\", instance=\"$instance\"})",
          "format": "time_series",
          "intervalFactor": 1,
          "legendFormat": "Load",
          "refId": "B"
        }
      ],
      "thresholds": [],
      "timeFrom": null,
      "timeShift": null,
      "title": "MetapartitionOperations",
      "tooltip": {
        "shared": true,
        "sort": 0,
        "value_type": "individual"
      },
      "type": "graph",
      "xaxis": {
        "buckets": null,
        "mode": "time",
        "name": null,
        "show": true,
        "values": []
      },
      "yaxes": [
        {
          "format": "ns",
          "label": null,
          "logBase": 1,
          "max": null,
          "min": null,
          "show": true
        },
        {
          "format": "short",
          "label": null,
          "logBase": 1,
          "max": null,
          "min": null,
          "show": true
        }
      ],
      "yaxis": {
        "align": false,
        "alignLevel": null
      }
    },
    {
      "aliasColors": {},
      "bars": false,
      "dashLength": 10,
      "dashes": false,
      "datasource": "default",
      "fill": 1,
      "gridPos": {
        "h": 6,
        "w": 7,
        "x": 7,
        "y": 29
      },
      "id": 75,
      "legend": {
        "avg": false,
        "current": false,
        "max": false,
        "min": false,
        "show": true,
        "total": false,
        "values": false
      },
      "lines": true,
      "linewidth": 1,
      "links": [],
      "nullPointMode": "null",
      "percentage": false,
      "pointradius": 5,
      "points": false,
      "renderer": "flot",
      "seriesOverrides": [],
      "spaceLength": 10,
      "stack": false,
      "steppedLine": false,
      "targets": [
        {
          "expr": "sum([[app]]_dataNode_OpLoadDataPartition{cluster=~\"$cluster\"})",
          "format": "time_series",
          "intervalFactor": 1,
          "legendFormat": "LoadDataPartition",
          "refId": "G"
        },
        {
          "expr": "sum([[app]]_dataNode_OpDataNodeHeartbeat{cluster=~\"$cluster\"})",
          "format": "time_series",
          "intervalFactor": 1,
          "legendFormat": "DataNodeHeartbeat",
          "refId": "F"
        },
        {
          "expr": "sum([[app]]_dataNode_OpCreateDataPartition{cluster=~\"$cluster\"})",
          "format": "time_series",
          "intervalFactor": 1,
          "legendFormat": "CreateDataPartition",
          "refId": "A"
        },
        {
          "expr": "sum([[app]]_dataNode_OpAddDataPartitionRaftMember{cluster=~\"$cluster\"})",
          "format": "time_series",
          "intervalFactor": 1,
          "legendFormat": "AddDataPartitionRaftMember",
          "refId": "B"
        },
        {
          "expr": "sum([[app]]_dataNode_OpDeleteDataPartition{cluster=~\"$cluster\"})",
          "format": "time_series",
          "intervalFactor": 1,
          "legendFormat": "DeleteDataPartition",
          "refId": "C"
        }
      ],
      "thresholds": [],
      "timeFrom": null,
      "timeShift": null,
      "title": "DatapartitionOperations",
      "tooltip": {
        "shared": true,
        "sort": 0,
        "value_type": "individual"
      },
      "type": "graph",
      "xaxis": {
        "buckets": null,
        "mode": "time",
        "name": null,
        "show": true,
        "values": []
      },
      "yaxes": [
        {
          "format": "ns",
          "label": null,
          "logBase": 1,
          "max": null,
          "min": null,
          "show": true
        },
        {
          "format": "short",
          "label": null,
          "logBase": 1,
          "max": null,
          "min": null,
          "show": true
        }
      ],
      "yaxis": {
        "align": false,
        "alignLevel": null
      }
    },
    {
      "collapsed": false,
      "gridPos": {
        "h": 1,
        "w": 24,
        "x": 0,
        "y": 35
      },
      "id": 34,
      "panels": [],
      "title": "Master",
      "type": "row"
    },
    {
      "aliasColors": {},
      "bars": true,
      "dashLength": 10,
      "dashes": false,
      "datasource": null,
      "fill": 1,
      "gridPos": {
        "h": 7,
        "w": 7,
        "x": 0,
        "y": 36
      },
      "id": 162,
      "legend": {
        "avg": false,
        "current": false,
        "max": false,
        "min": false,
        "show": true,
        "total": false,
        "values": false
      },
      "lines": false,
      "linewidth": 1,
      "links": [],
      "nullPointMode": "null",
      "percentage": false,
      "pointradius": 5,
      "points": false,
      "renderer": "flot",
      "seriesOverrides": [
        {
          "alias": "master_count",
          "yaxis": 2
        }
      ],
      "spaceLength": 10,
      "stack": false,
      "steppedLine": false,
      "targets": [
        {
          "expr": "go_info{cluster=\"$cluster\", role=\"master\"}",
          "format": "time_series",
          "hide": false,
          "intervalFactor": 1,
          "legendFormat": "[[cluster]]_master_{{instance}}",
          "refId": "C"
        },
        {
          "expr": "count(go_info{cluster=~\"$cluster\", role=~\"master\"})",
          "format": "time_series",
          "hide": false,
          "intervalFactor": 1,
          "legendFormat": "[[cluster]]_master_count",
          "refId": "G"
        }
      ],
      "thresholds": [],
      "timeFrom": null,
      "timeShift": null,
      "title": "[[role]]_nodes",
      "tooltip": {
        "shared": true,
        "sort": 0,
        "value_type": "individual"
      },
      "type": "graph",
      "xaxis": {
        "buckets": null,
        "mode": "time",
        "name": null,
        "show": true,
        "values": []
      },
      "yaxes": [
        {
          "format": "locale",
          "label": null,
          "logBase": 1,
          "max": null,
          "min": null,
          "show": true
        },
        {
          "format": "locale",
          "label": null,
          "logBase": 1,
          "max": null,
          "min": null,
          "show": true
        }
      ],
      "yaxis": {
        "align": false,
        "alignLevel": null
      }
    },
    {
      "aliasColors": {},
      "bars": true,
      "dashLength": 10,
      "dashes": false,
      "datasource": null,
      "fill": 1,
      "gridPos": {
        "h": 7,
        "w": 7,
        "x": 7,
        "y": 36
      },
      "id": 163,
      "legend": {
        "avg": false,
        "current": false,
        "max": false,
        "min": false,
        "show": true,
        "total": false,
        "values": false
      },
      "lines": false,
      "linewidth": 1,
      "links": [],
      "nullPointMode": "null",
      "percentage": false,
      "pointradius": 5,
      "points": false,
      "renderer": "flot",
      "seriesOverrides": [],
      "spaceLength": 10,
      "stack": true,
      "steppedLine": false,
      "targets": [
        {
          "expr": "go_info{cluster=\"$cluster\", role=\"metanode\"}",
          "format": "time_series",
          "intervalFactor": 1,
          "legendFormat": "meta_{{instance}}",
          "refId": "C"
        },
        {
          "expr": "sum([[app]]_master_metaNodes_count{cluster=~\"$cluster\"})",
          "format": "time_series",
          "hide": true,
          "intervalFactor": 1,
          "legendFormat": "metaNode",
          "refId": "B"
        },
        {
          "expr": "sum([[app]]_master_dataNodes_count{cluster=~\"$cluster\"})",
          "format": "time_series",
          "hide": true,
          "intervalFactor": 1,
          "legendFormat": "datanode",
          "refId": "A"
        },
        {
          "expr": "[[app]]_master_metaNodes_count",
          "format": "time_series",
          "hide": true,
          "intervalFactor": 1,
          "legendFormat": "meta_count",
          "refId": "D"
        },
        {
          "expr": "[[app]]_master_dataNodes_count",
          "format": "time_series",
          "hide": true,
          "intervalFactor": 1,
          "legendFormat": "datanode_count",
          "refId": "E"
        },
        {
          "expr": "count(process_start_time_seconds{role=\"fuseclient\"})",
          "format": "time_series",
          "hide": true,
          "intervalFactor": 1,
          "legendFormat": "client_count",
          "refId": "F"
        }
      ],
      "thresholds": [],
      "timeFrom": null,
      "timeShift": null,
      "title": "[[cluster]]_metanode_alive",
      "tooltip": {
        "shared": true,
        "sort": 0,
        "value_type": "individual"
      },
      "type": "graph",
      "xaxis": {
        "buckets": null,
        "mode": "time",
        "name": null,
        "show": true,
        "values": []
      },
      "yaxes": [
        {
          "format": "locale",
          "label": null,
          "logBase": 1,
          "max": null,
          "min": null,
          "show": true
        },
        {
          "format": "short",
          "label": null,
          "logBase": 1,
          "max": null,
          "min": null,
          "show": true
        }
      ],
      "yaxis": {
        "align": false,
        "alignLevel": null
      }
    },
    {
      "aliasColors": {},
      "bars": true,
      "dashLength": 10,
      "dashes": false,
      "datasource": null,
      "fill": 1,
      "gridPos": {
        "h": 7,
        "w": 7,
        "x": 0,
        "y": 43
      },
      "id": 164,
      "legend": {
        "avg": false,
        "current": false,
        "max": false,
        "min": false,
        "show": true,
        "total": false,
        "values": false
      },
      "lines": false,
      "linewidth": 1,
      "links": [],
      "nullPointMode": "null",
      "percentage": false,
      "pointradius": 5,
      "points": false,
      "renderer": "flot",
      "seriesOverrides": [],
      "spaceLength": 10,
      "stack": true,
      "steppedLine": false,
      "targets": [
        {
          "expr": "go_info{cluster=\"$cluster\", role=\"dataNode\"}",
          "format": "time_series",
          "intervalFactor": 1,
          "legendFormat": "{{instance}}",
          "refId": "C"
        },
        {
          "expr": "sum([[app]]_master_metaNodes_count{cluster=~\"$cluster\"})",
          "format": "time_series",
          "hide": true,
          "intervalFactor": 1,
          "legendFormat": "metaNode",
          "refId": "B"
        },
        {
          "expr": "sum([[app]]_master_dataNodes_count{cluster=~\"$cluster\"})",
          "format": "time_series",
          "hide": true,
          "intervalFactor": 1,
          "legendFormat": "datanode",
          "refId": "A"
        },
        {
          "expr": "[[app]]_master_metaNodes_count",
          "format": "time_series",
          "hide": true,
          "intervalFactor": 1,
          "legendFormat": "meta_count",
          "refId": "D"
        },
        {
          "expr": "[[app]]_master_dataNodes_count",
          "format": "time_series",
          "hide": true,
          "intervalFactor": 1,
          "legendFormat": "datanode_count",
          "refId": "E"
        },
        {
          "expr": "count(process_start_time_seconds{role=\"fuseclient\"})",
          "format": "time_series",
          "hide": true,
          "intervalFactor": 1,
          "legendFormat": "client_count",
          "refId": "F"
        }
      ],
      "thresholds": [],
      "timeFrom": null,
      "timeShift": null,
      "title": "[[cluster]]_datanode_alive",
      "tooltip": {
        "shared": true,
        "sort": 0,
        "value_type": "individual"
      },
      "type": "graph",
      "xaxis": {
        "buckets": null,
        "mode": "time",
        "name": null,
        "show": true,
        "values": []
      },
      "yaxes": [
        {
          "format": "locale",
          "label": null,
          "logBase": 1,
          "max": null,
          "min": null,
          "show": true
        },
        {
          "format": "short",
          "label": null,
          "logBase": 1,
          "max": null,
          "min": null,
          "show": true
        }
      ],
      "yaxis": {
        "align": false,
        "alignLevel": null
      }
    },
    {
      "aliasColors": {},
      "bars": true,
      "dashLength": 10,
      "dashes": false,
      "datasource": null,
      "fill": 1,
      "gridPos": {
        "h": 7,
        "w": 7,
        "x": 7,
        "y": 43
      },
      "id": 165,
      "legend": {
        "avg": false,
        "current": false,
        "max": false,
        "min": false,
        "show": true,
        "total": false,
        "values": false
      },
      "lines": false,
      "linewidth": 1,
      "links": [],
      "nullPointMode": "null",
      "percentage": false,
      "pointradius": 5,
      "points": false,
      "renderer": "flot",
      "seriesOverrides": [],
      "spaceLength": 10,
      "stack": true,
      "steppedLine": false,
      "targets": [
        {
          "expr": "go_info{cluster=\"$cluster\", role=\"fuseclient\"}",
          "format": "time_series",
          "intervalFactor": 1,
          "legendFormat": "fuseclient_{{instance}}",
          "refId": "C"
        },
        {
          "expr": "sum([[app]]_master_metaNodes_count{cluster=~\"$cluster\"})",
          "format": "time_series",
          "hide": true,
          "intervalFactor": 1,
          "legendFormat": "metaNode",
          "refId": "B"
        },
        {
          "expr": "sum([[app]]_master_dataNodes_count{cluster=~\"$cluster\"})",
          "format": "time_series",
          "hide": true,
          "intervalFactor": 1,
          "legendFormat": "datanode",
          "refId": "A"
        },
        {
          "expr": "[[app]]_master_metaNodes_count",
          "format": "time_series",
          "hide": true,
          "intervalFactor": 1,
          "legendFormat": "meta_count",
          "refId": "D"
        },
        {
          "expr": "[[app]]_master_dataNodes_count",
          "format": "time_series",
          "hide": true,
          "intervalFactor": 1,
          "legendFormat": "datanode_count",
          "refId": "E"
        },
        {
          "expr": "count(process_start_time_seconds{role=\"fuseclient\"})",
          "format": "time_series",
          "hide": true,
          "intervalFactor": 1,
          "legendFormat": "client_count",
          "refId": "F"
        }
      ],
      "thresholds": [],
      "timeFrom": null,
      "timeShift": null,
      "title": "[[cluster]]_fuseclient_alive",
      "tooltip": {
        "shared": true,
        "sort": 0,
        "value_type": "individual"
      },
      "type": "graph",
      "xaxis": {
        "buckets": null,
        "mode": "time",
        "name": null,
        "show": true,
        "values": []
      },
      "yaxes": [
        {
          "format": "locale",
          "label": null,
          "logBase": 1,
          "max": null,
          "min": null,
          "show": true
        },
        {
          "format": "short",
          "label": null,
          "logBase": 1,
          "max": null,
          "min": null,
          "show": true
        }
      ],
      "yaxis": {
        "align": false,
        "alignLevel": null
      }
    },
    {
      "collapsed": true,
      "gridPos": {
        "h": 1,
        "w": 24,
        "x": 0,
        "y": 50
      },
      "id": 36,
      "panels": [
        {
          "aliasColors": {},
          "bars": false,
          "dashLength": 10,
          "dashes": false,
          "datasource": null,
          "fill": 1,
          "gridPos": {
            "h": 8,
            "w": 8,
            "x": 0,
            "y": 3
          },
          "id": 117,
          "legend": {
            "avg": false,
            "current": false,
            "max": false,
            "min": false,
            "show": true,
            "total": false,
            "values": false
          },
          "lines": true,
          "linewidth": 1,
          "links": [],
          "nullPointMode": "null",
          "percentage": false,
          "pointradius": 5,
          "points": false,
          "renderer": "flot",
          "seriesOverrides": [
            {
              "alias": "/.*_ops_.*/",
              "yaxis": 2
            }
          ],
          "spaceLength": 10,
          "stack": false,
          "steppedLine": false,
          "targets": [
            {
              "expr": "[[app]]_metanode_OpMetaCreateInode{app=~\"$app\", cluster=\"$cluster\"}",
              "format": "time_series",
              "hide": false,
              "intervalFactor": 1,
              "legendFormat": "OpMetaCreateInode_{{instance}}",
              "refId": "B"
            },
            {
              "expr": "rate([[app]]_metanode_OpMetaCreateInode_count{app=~\"$app\", cluster=\"$cluster\"}[1m])",
              "format": "time_series",
              "hide": false,
              "intervalFactor": 1,
              "legendFormat": "OpMetaCreateInode_ops_{{instance}}",
              "refId": "A"
            }
          ],
          "thresholds": [],
          "timeFrom": null,
          "timeShift": null,
          "title": "InodeCreate",
          "tooltip": {
            "shared": true,
            "sort": 0,
            "value_type": "individual"
          },
          "type": "graph",
          "xaxis": {
            "buckets": null,
            "mode": "time",
            "name": null,
            "show": true,
            "values": []
          },
          "yaxes": [
            {
              "format": "ns",
              "label": null,
              "logBase": 1,
              "max": null,
              "min": null,
              "show": true
            },
            {
              "format": "ops",
              "label": null,
              "logBase": 1,
              "max": null,
              "min": null,
              "show": true
            }
          ],
          "yaxis": {
            "align": false,
            "alignLevel": null
          }
        },
        {
          "aliasColors": {},
          "bars": false,
          "dashLength": 10,
          "dashes": false,
          "datasource": null,
          "fill": 1,
          "gridPos": {
            "h": 8,
            "w": 8,
            "x": 8,
            "y": 3
          },
          "id": 44,
          "legend": {
            "avg": false,
            "current": false,
            "max": false,
            "min": false,
            "show": true,
            "total": false,
            "values": false
          },
          "lines": true,
          "linewidth": 1,
          "links": [],
          "nullPointMode": "null",
          "percentage": false,
          "pointradius": 5,
          "points": false,
          "renderer": "flot",
          "seriesOverrides": [],
          "spaceLength": 10,
          "stack": false,
          "steppedLine": false,
          "targets": [
            {
              "expr": "[[app]]_metanode_OpMetaBatchInodeGet{app=~\"$app\", cluster=\"$cluster\"}",
              "format": "time_series",
              "intervalFactor": 1,
              "legendFormat": "OpMetaBatchInodeGet_{{instance}}",
              "refId": "A"
            },
            {
              "expr": "[[app]]_metanode_OpMetaCreateInode{app=~\"$app\", cluster=\"$cluster\"}",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "OpMetaCreateInode",
              "refId": "B"
            },
            {
              "expr": "[[app]]_metanode_OpMetaDeleteInode{app=~\"$app\", cluster=\"$cluster\"}",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "OpMetaDeleteInode_{{instance}}",
              "refId": "C"
            },
            {
              "expr": "[[app]]_metanode_OpMetaEvictInode{app=~\"$app\", cluster=\"$cluster\"}",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "OpMetaEvictInode_{{instance}}",
              "refId": "D"
            },
            {
              "expr": "[[app]]_metanode_OpMetaInodeGet{app=~\"$app\", cluster=\"$cluster\"}",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "OpMetaInodeGet_{{instance}}",
              "refId": "E"
            },
            {
              "expr": "[[app]]_metanode_OpMetaLinkInode{app=~\"$app\", cluster=\"$cluster\"}",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "OpMetaLinkInode_{{instance}}",
              "refId": "F"
            }
          ],
          "thresholds": [],
          "timeFrom": null,
          "timeShift": null,
          "title": "InodeBatchGet",
          "tooltip": {
            "shared": true,
            "sort": 0,
            "value_type": "individual"
          },
          "type": "graph",
          "xaxis": {
            "buckets": null,
            "mode": "time",
            "name": null,
            "show": true,
            "values": []
          },
          "yaxes": [
            {
              "format": "ns",
              "label": null,
              "logBase": 1,
              "max": null,
              "min": null,
              "show": true
            },
            {
              "format": "short",
              "label": null,
              "logBase": 1,
              "max": null,
              "min": null,
              "show": true
            }
          ],
          "yaxis": {
            "align": false,
            "alignLevel": null
          }
        },
        {
          "aliasColors": {},
          "bars": false,
          "dashLength": 10,
          "dashes": false,
          "datasource": null,
          "fill": 1,
          "gridPos": {
            "h": 8,
            "w": 8,
            "x": 16,
            "y": 3
          },
          "id": 119,
          "legend": {
            "avg": false,
            "current": false,
            "max": false,
            "min": false,
            "show": true,
            "total": false,
            "values": false
          },
          "lines": true,
          "linewidth": 1,
          "links": [],
          "nullPointMode": "null",
          "percentage": false,
          "pointradius": 5,
          "points": false,
          "renderer": "flot",
          "seriesOverrides": [
            {
              "alias": "/.*_ops_.*/",
              "yaxis": 2
            }
          ],
          "spaceLength": 10,
          "stack": false,
          "steppedLine": false,
          "targets": [
            {
              "expr": "[[app]]_metanode_OpMetaBatchInodeGet{app=~\"$app\", cluster=\"$cluster\"}",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "OpMetaBatchInodeGet_{{instance}}",
              "refId": "A"
            },
            {
              "expr": "[[app]]_metanode_OpMetaCreateInode{app=~\"$app\", cluster=\"$cluster\"}",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "OpMetaCreateInode",
              "refId": "B"
            },
            {
              "expr": "[[app]]_metanode_OpMetaDeleteInode{app=~\"$app\", cluster=\"$cluster\"}",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "OpMetaDeleteInode_{{instance}}",
              "refId": "C"
            },
            {
              "expr": "[[app]]_metanode_OpMetaEvictInode{app=~\"$app\", cluster=\"$cluster\"}",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "OpMetaEvictInode_{{instance}}",
              "refId": "D"
            },
            {
              "expr": "[[app]]_metanode_OpMetaInodeGet{app=~\"$app\", cluster=\"$cluster\"}",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "OpMetaInodeGet_{{instance}}",
              "refId": "E"
            },
            {
              "expr": "[[app]]_metanode_OpMetaLinkInode{app=~\"$app\", cluster=\"$cluster\"}",
              "format": "time_series",
              "hide": false,
              "intervalFactor": 1,
              "legendFormat": "OpMetaLinkInode_{{instance}}",
              "refId": "F"
            },
            {
              "expr": "rate([[app]]_metanode_OpMetaLinkInode_count{app=~\"$app\", cluster=\"$cluster\"}[1m])",
              "format": "time_series",
              "hide": false,
              "intervalFactor": 1,
              "legendFormat": "OpMetaLinkInode_ops_{{instance}}",
              "refId": "G"
            }
          ],
          "thresholds": [],
          "timeFrom": null,
          "timeShift": null,
          "title": "InodeLink",
          "tooltip": {
            "shared": true,
            "sort": 0,
            "value_type": "individual"
          },
          "type": "graph",
          "xaxis": {
            "buckets": null,
            "mode": "time",
            "name": null,
            "show": true,
            "values": []
          },
          "yaxes": [
            {
              "format": "ns",
              "label": null,
              "logBase": 1,
              "max": null,
              "min": null,
              "show": true
            },
            {
              "format": "ops",
              "label": null,
              "logBase": 1,
              "max": null,
              "min": null,
              "show": true
            }
          ],
          "yaxis": {
            "align": false,
            "alignLevel": null
          }
        },
        {
          "aliasColors": {},
          "bars": false,
          "dashLength": 10,
          "dashes": false,
          "datasource": null,
          "fill": 1,
          "gridPos": {
            "h": 8,
            "w": 8,
            "x": 0,
            "y": 11
          },
          "id": 120,
          "legend": {
            "avg": false,
            "current": false,
            "max": false,
            "min": false,
            "show": true,
            "total": false,
            "values": false
          },
          "lines": true,
          "linewidth": 1,
          "links": [],
          "nullPointMode": "null",
          "percentage": false,
          "pointradius": 5,
          "points": false,
          "renderer": "flot",
          "seriesOverrides": [],
          "spaceLength": 10,
          "stack": false,
          "steppedLine": false,
          "targets": [
            {
              "expr": "[[app]]_metanode_OpMetaBatchInodeGet{app=~\"$app\", cluster=\"$cluster\"}",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "OpMetaBatchInodeGet_{{instance}}",
              "refId": "A"
            },
            {
              "expr": "[[app]]_metanode_OpMetaCreateInode{app=~\"$app\", cluster=\"$cluster\"}",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "OpMetaCreateInode",
              "refId": "B"
            },
            {
              "expr": "[[app]]_metanode_OpMetaDeleteInode{app=~\"$app\", cluster=\"$cluster\"}",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "OpMetaDeleteInode_{{instance}}",
              "refId": "C"
            },
            {
              "expr": "[[app]]_metanode_OpMetaEvictInode{app=~\"$app\", cluster=\"$cluster\"}",
              "format": "time_series",
              "hide": false,
              "intervalFactor": 1,
              "legendFormat": "OpMetaEvictInode_{{instance}}",
              "refId": "D"
            },
            {
              "expr": "[[app]]_metanode_OpMetaInodeGet{app=~\"$app\", cluster=\"$cluster\"}",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "OpMetaInodeGet_{{instance}}",
              "refId": "E"
            },
            {
              "expr": "[[app]]_metanode_OpMetaLinkInode{app=~\"$app\", cluster=\"$cluster\"}",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "OpMetaLinkInode_{{instance}}",
              "refId": "F"
            }
          ],
          "thresholds": [],
          "timeFrom": null,
          "timeShift": null,
          "title": "InodeEvict",
          "tooltip": {
            "shared": true,
            "sort": 0,
            "value_type": "individual"
          },
          "type": "graph",
          "xaxis": {
            "buckets": null,
            "mode": "time",
            "name": null,
            "show": true,
            "values": []
          },
          "yaxes": [
            {
              "format": "ns",
              "label": null,
              "logBase": 1,
              "max": null,
              "min": null,
              "show": true
            },
            {
              "format": "short",
              "label": null,
              "logBase": 1,
              "max": null,
              "min": null,
              "show": true
            }
          ],
          "yaxis": {
            "align": false,
            "alignLevel": null
          }
        },
        {
          "aliasColors": {},
          "bars": false,
          "dashLength": 10,
          "dashes": false,
          "datasource": null,
          "description": "",
          "fill": 1,
          "gridPos": {
            "h": 8,
            "w": 8,
            "x": 8,
            "y": 11
          },
          "id": 121,
          "legend": {
            "avg": false,
            "current": false,
            "max": false,
            "min": false,
            "show": true,
            "total": false,
            "values": false
          },
          "lines": true,
          "linewidth": 1,
          "links": [],
          "nullPointMode": "null",
          "percentage": false,
          "pointradius": 5,
          "points": false,
          "renderer": "flot",
          "seriesOverrides": [],
          "spaceLength": 10,
          "stack": false,
          "steppedLine": false,
          "targets": [
            {
              "expr": "[[app]]_metanode_OpCreateMetaPartition{app=\"$app\", cluster=\"$cluster\"}",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "OpCreateMetaPartition_{{instance}}",
              "refId": "A"
            },
            {
              "expr": "[[app]]_metanode_OpLoadMetaPartition{app=\"$app\", cluster=\"$cluster\"}",
              "format": "time_series",
              "hide": false,
              "intervalFactor": 1,
              "legendFormat": "OpLoadMetaPartition_{{instance}}",
              "refId": "B"
            }
          ],
          "thresholds": [],
          "timeFrom": null,
          "timeShift": null,
          "title": "LoadMetaPartition",
          "tooltip": {
            "shared": true,
            "sort": 0,
            "value_type": "individual"
          },
          "type": "graph",
          "xaxis": {
            "buckets": null,
            "mode": "time",
            "name": null,
            "show": true,
            "values": []
          },
          "yaxes": [
            {
              "format": "ns",
              "label": null,
              "logBase": 1,
              "max": null,
              "min": null,
              "show": true
            },
            {
              "format": "short",
              "label": null,
              "logBase": 1,
              "max": null,
              "min": null,
              "show": true
            }
          ],
          "yaxis": {
            "align": false,
            "alignLevel": null
          }
        },
        {
          "aliasColors": {},
          "bars": false,
          "dashLength": 10,
          "dashes": false,
          "datasource": null,
          "fill": 1,
          "gridPos": {
            "h": 8,
            "w": 8,
            "x": 16,
            "y": 11
          },
          "id": 58,
          "legend": {
            "avg": false,
            "current": false,
            "max": false,
            "min": false,
            "show": true,
            "total": false,
            "values": false
          },
          "lines": true,
          "linewidth": 1,
          "links": [],
          "nullPointMode": "null",
          "percentage": false,
          "pointradius": 5,
          "points": false,
          "renderer": "flot",
          "seriesOverrides": [],
          "spaceLength": 10,
          "stack": false,
          "steppedLine": false,
          "targets": [
            {
              "expr": "[[app]]_metanode_OpCreateMetaPartition{app=\"$app\", cluster=\"$cluster\"}",
              "format": "time_series",
              "intervalFactor": 1,
              "legendFormat": "OpCreateMetaPartition_{{instance}}",
              "refId": "A"
            },
            {
              "expr": "[[app]]_metanode_OpLoadMetaPartition{app=\"$app\", cluster=\"$cluster\"}",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "OpLoadMetaPartition_{{instance}}",
              "refId": "B"
            }
          ],
          "thresholds": [],
          "timeFrom": null,
          "timeShift": null,
          "title": "CreateMetaPartition",
          "tooltip": {
            "shared": true,
            "sort": 0,
            "value_type": "individual"
          },
          "type": "graph",
          "xaxis": {
            "buckets": null,
            "mode": "time",
            "name": null,
            "show": true,
            "values": []
          },
          "yaxes": [
            {
              "format": "ns",
              "label": null,
              "logBase": 1,
              "max": null,
              "min": null,
              "show": true
            },
            {
              "format": "short",
              "label": null,
              "logBase": 1,
              "max": null,
              "min": null,
              "show": true
            }
          ],
          "yaxis": {
            "align": false,
            "alignLevel": null
          }
        },
        {
          "aliasColors": {},
          "bars": false,
          "dashLength": 10,
          "dashes": false,
          "datasource": null,
          "fill": 1,
          "gridPos": {
            "h": 8,
            "w": 8,
            "x": 0,
            "y": 19
          },
          "id": 88,
          "legend": {
            "avg": false,
            "current": false,
            "max": false,
            "min": false,
            "show": true,
            "total": false,
            "values": false
          },
          "lines": true,
          "linewidth": 1,
          "links": [],
          "nullPointMode": "null",
          "percentage": false,
          "pointradius": 5,
          "points": false,
          "renderer": "flot",
          "seriesOverrides": [],
          "spaceLength": 10,
          "stack": false,
          "steppedLine": false,
          "targets": [
            {
              "expr": "[[app]]_metanode_OpMetaExtentsList{app=~\"$app\", cluster=\"$cluster\", instance=\"$instance\"}",
              "format": "time_series",
              "intervalFactor": 1,
              "legendFormat": "OpMetaExtentsList_{{instance}}",
              "refId": "B"
            }
          ],
          "thresholds": [],
          "timeFrom": null,
          "timeShift": null,
          "title": "MetaExtentsList",
          "tooltip": {
            "shared": true,
            "sort": 0,
            "value_type": "individual"
          },
          "type": "graph",
          "xaxis": {
            "buckets": null,
            "mode": "time",
            "name": null,
            "show": true,
            "values": []
          },
          "yaxes": [
            {
              "format": "ns",
              "label": null,
              "logBase": 1,
              "max": null,
              "min": null,
              "show": true
            },
            {
              "format": "short",
              "label": null,
              "logBase": 1,
              "max": null,
              "min": null,
              "show": true
            }
          ],
          "yaxis": {
            "align": false,
            "alignLevel": null
          }
        },
        {
          "aliasColors": {},
          "bars": false,
          "dashLength": 10,
          "dashes": false,
          "datasource": null,
          "fill": 1,
          "gridPos": {
            "h": 8,
            "w": 8,
            "x": 8,
            "y": 19
          },
          "id": 50,
          "legend": {
            "avg": false,
            "current": false,
            "max": false,
            "min": false,
            "show": true,
            "total": false,
            "values": false
          },
          "lines": true,
          "linewidth": 1,
          "links": [],
          "nullPointMode": "null",
          "percentage": false,
          "pointradius": 5,
          "points": false,
          "renderer": "flot",
          "seriesOverrides": [],
          "spaceLength": 10,
          "stack": false,
          "steppedLine": false,
          "targets": [
            {
              "expr": "[[app]]_metanode_OpMetaExtentsAdd{app=~\"$app\", cluster=\"$cluster\"}",
              "format": "time_series",
              "intervalFactor": 1,
              "legendFormat": "OpMetaExtentsAdd_{{instance}}",
              "refId": "A"
            },
            {
              "expr": "[[app]]_metanode_OpMetaExtentsList{app=~\"$app\", cluster=\"$cluster\"}",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "OpMetaExtentsList_{{instance}}",
              "refId": "B"
            }
          ],
          "thresholds": [],
          "timeFrom": null,
          "timeShift": null,
          "title": "MetaExtentsAdd",
          "tooltip": {
            "shared": true,
            "sort": 0,
            "value_type": "individual"
          },
          "type": "graph",
          "xaxis": {
            "buckets": null,
            "mode": "time",
            "name": null,
            "show": true,
            "values": []
          },
          "yaxes": [
            {
              "format": "ns",
              "label": null,
              "logBase": 1,
              "max": null,
              "min": null,
              "show": true
            },
            {
              "format": "short",
              "label": null,
              "logBase": 1,
              "max": null,
              "min": null,
              "show": true
            }
          ],
          "yaxis": {
            "align": false,
            "alignLevel": null
          }
        },
        {
          "aliasColors": {},
          "bars": false,
          "dashLength": 10,
          "dashes": false,
          "datasource": null,
          "fill": 1,
          "gridPos": {
            "h": 8,
            "w": 8,
            "x": 16,
            "y": 19
          },
          "id": 126,
          "legend": {
            "avg": false,
            "current": false,
            "max": false,
            "min": false,
            "show": true,
            "total": false,
            "values": false
          },
          "lines": true,
          "linewidth": 1,
          "links": [],
          "nullPointMode": "null",
          "percentage": false,
          "pointradius": 5,
          "points": false,
          "renderer": "flot",
          "seriesOverrides": [],
          "spaceLength": 10,
          "stack": false,
          "steppedLine": false,
          "targets": [
            {
              "expr": "[[app]]_metanode_OpMetaNodeHeartbeat{app=~\"$app\", cluster=\"$cluster\"}",
              "format": "time_series",
              "hide": false,
              "intervalFactor": 1,
              "legendFormat": "OpMetaNodeHeartbeat_{{instance}}",
              "refId": "C"
            }
          ],
          "thresholds": [],
          "timeFrom": null,
          "timeShift": null,
          "title": "MetaNodeHeartbeat",
          "tooltip": {
            "shared": true,
            "sort": 0,
            "value_type": "individual"
          },
          "type": "graph",
          "xaxis": {
            "buckets": null,
            "mode": "time",
            "name": null,
            "show": true,
            "values": []
          },
          "yaxes": [
            {
              "format": "ns",
              "label": null,
              "logBase": 1,
              "max": null,
              "min": null,
              "show": true
            },
            {
              "format": "short",
              "label": null,
              "logBase": 1,
              "max": null,
              "min": null,
              "show": true
            }
          ],
          "yaxis": {
            "align": false,
            "alignLevel": null
          }
        },
        {
          "aliasColors": {},
          "bars": false,
          "dashLength": 10,
          "dashes": false,
          "datasource": null,
          "fill": 1,
          "gridPos": {
            "h": 8,
            "w": 8,
            "x": 0,
            "y": 27
          },
          "id": 72,
          "legend": {
            "avg": false,
            "current": false,
            "max": false,
            "min": false,
            "show": true,
            "total": false,
            "values": false
          },
          "lines": true,
          "linewidth": 1,
          "links": [],
          "nullPointMode": "null",
          "percentage": false,
          "pointradius": 5,
          "points": false,
          "renderer": "flot",
          "seriesOverrides": [],
          "spaceLength": 10,
          "stack": false,
          "steppedLine": false,
          "targets": [
            {
              "expr": "[[app]]_metanode_OpMetaCreateDentry{app=\"$app\",cluster=~\"$cluster\"}",
              "format": "time_series",
              "intervalFactor": 1,
              "legendFormat": "OpMetaCreateDentry_{{instance}}",
              "refId": "A"
            },
            {
              "expr": "[[app]]_metanode_OpMetaDeleteDentry{app=\"$app\",cluster=~\"$cluster\"}",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "OpMetaDeleteDentry_{{instance}}",
              "refId": "B"
            },
            {
              "expr": "[[app]]_metanode_OpMetaUpdateDentry{app=\"$app\",cluster=~\"$cluster\"}",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "OpMetaUpdateDentry_{{instance}}",
              "refId": "C"
            }
          ],
          "thresholds": [],
          "timeFrom": null,
          "timeShift": null,
          "title": "MetaCreateDentry",
          "tooltip": {
            "shared": true,
            "sort": 0,
            "value_type": "individual"
          },
          "type": "graph",
          "xaxis": {
            "buckets": null,
            "mode": "time",
            "name": null,
            "show": true,
            "values": []
          },
          "yaxes": [
            {
              "format": "ns",
              "label": null,
              "logBase": 1,
              "max": null,
              "min": null,
              "show": true
            },
            {
              "format": "short",
              "label": null,
              "logBase": 1,
              "max": null,
              "min": null,
              "show": true
            }
          ],
          "yaxis": {
            "align": false,
            "alignLevel": null
          }
        },
        {
          "aliasColors": {},
          "bars": false,
          "dashLength": 10,
          "dashes": false,
          "datasource": null,
          "fill": 1,
          "gridPos": {
            "h": 8,
            "w": 8,
            "x": 8,
            "y": 27
          },
          "id": 125,
          "legend": {
            "avg": false,
            "current": false,
            "max": false,
            "min": false,
            "show": true,
            "total": false,
            "values": false
          },
          "lines": true,
          "linewidth": 1,
          "links": [],
          "nullPointMode": "null",
          "percentage": false,
          "pointradius": 5,
          "points": false,
          "renderer": "flot",
          "seriesOverrides": [],
          "spaceLength": 10,
          "stack": false,
          "steppedLine": false,
          "targets": [
            {
              "expr": "[[app]]_metanode_OpMetaOpen{app=~\"$app\", cluster=\"$cluster\"}",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "OpMetaOpen_{{instance}}",
              "refId": "A"
            },
            {
              "expr": "[[app]]_metanode_OpMetaLookup{app=~\"$app\", cluster=\"$cluster\", instance=\"$instance\"}",
              "format": "time_series",
              "hide": false,
              "intervalFactor": 1,
              "legendFormat": "OpMetaLookup_{{instance}}",
              "refId": "B"
            },
            {
              "expr": "[[app]]_metanode_OpMetaNodeHeartbeat{app=~\"$app\", cluster=\"$cluster\"}",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "OpMetaNodeHeartbeat_{{instance}}",
              "refId": "C"
            },
            {
              "expr": "[[app]]_metanode_OpMetaReadDir{app=~\"$app\", cluster=\"$cluster\"}",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "OpMetaReadDir_{{instance}}",
              "refId": "D"
            },
            {
              "expr": "[[app]]_metanode_OpMetaReleaseOpen{app=~\"$app\", cluster=\"$cluster\"}",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "OpMetaReleaseOpen_{{instance}}",
              "refId": "E"
            },
            {
              "expr": "[[app]]_metanode_OpMetaSetattr{app=~\"$app\", cluster=\"$cluster\"}",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "OpMetaSetattr_{{instance}}",
              "refId": "F"
            },
            {
              "expr": "[[app]]_metanode_OpMetaTruncate{app=~\"$app\", cluster=\"$cluster\"}",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "OpMetaTruncate_{{instance}}",
              "refId": "G"
            }
          ],
          "thresholds": [],
          "timeFrom": null,
          "timeShift": null,
          "title": "MetaLookup",
          "tooltip": {
            "shared": true,
            "sort": 0,
            "value_type": "individual"
          },
          "type": "graph",
          "xaxis": {
            "buckets": null,
            "mode": "time",
            "name": null,
            "show": true,
            "values": []
          },
          "yaxes": [
            {
              "format": "ns",
              "label": null,
              "logBase": 1,
              "max": null,
              "min": null,
              "show": true
            },
            {
              "format": "short",
              "label": null,
              "logBase": 1,
              "max": null,
              "min": null,
              "show": true
            }
          ],
          "yaxis": {
            "align": false,
            "alignLevel": null
          }
        },
        {
          "aliasColors": {},
          "bars": false,
          "dashLength": 10,
          "dashes": false,
          "datasource": null,
          "fill": 1,
          "gridPos": {
            "h": 8,
            "w": 8,
            "x": 16,
            "y": 27
          },
          "id": 122,
          "legend": {
            "avg": false,
            "current": false,
            "max": false,
            "min": false,
            "show": true,
            "total": false,
            "values": false
          },
          "lines": true,
          "linewidth": 1,
          "links": [],
          "nullPointMode": "null",
          "percentage": false,
          "pointradius": 5,
          "points": false,
          "renderer": "flot",
          "seriesOverrides": [],
          "spaceLength": 10,
          "stack": false,
          "steppedLine": false,
          "targets": [
            {
              "expr": "[[app]]_metanode_OpMetaCreateDentry{app=\"$app\",cluster=~\"$cluster\", instance=\"$instance\"}",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "OpMetaCreateDentry_{{instance}}",
              "refId": "A"
            },
            {
              "expr": "[[app]]_metanode_OpMetaDeleteDentry{app=\"$app\",cluster=~\"$cluster\", instance=\"$instance\"}",
              "format": "time_series",
              "hide": false,
              "intervalFactor": 1,
              "legendFormat": "OpMetaDeleteDentry_{{instance}}",
              "refId": "B"
            },
            {
              "expr": "[[app]]_metanode_OpMetaUpdateDentry{app=\"$app\",cluster=~\"$cluster\", instance=\"$instance\"}",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "OpMetaUpdateDentry_{{instance}}",
              "refId": "C"
            }
          ],
          "thresholds": [],
          "timeFrom": null,
          "timeShift": null,
          "title": "MetaDeleteDentry",
          "tooltip": {
            "shared": true,
            "sort": 0,
            "value_type": "individual"
          },
          "type": "graph",
          "xaxis": {
            "buckets": null,
            "mode": "time",
            "name": null,
            "show": true,
            "values": []
          },
          "yaxes": [
            {
              "format": "ns",
              "label": null,
              "logBase": 1,
              "max": null,
              "min": null,
              "show": true
            },
            {
              "format": "short",
              "label": null,
              "logBase": 1,
              "max": null,
              "min": null,
              "show": true
            }
          ],
          "yaxis": {
            "align": false,
            "alignLevel": null
          }
        },
        {
          "aliasColors": {},
          "bars": false,
          "dashLength": 10,
          "dashes": false,
          "datasource": null,
          "fill": 1,
          "gridPos": {
            "h": 8,
            "w": 8,
            "x": 0,
            "y": 35
          },
          "id": 124,
          "legend": {
            "avg": false,
            "current": false,
            "max": false,
            "min": false,
            "show": true,
            "total": false,
            "values": false
          },
          "lines": true,
          "linewidth": 1,
          "links": [],
          "nullPointMode": "null",
          "percentage": false,
          "pointradius": 5,
          "points": false,
          "renderer": "flot",
          "seriesOverrides": [
            {
              "alias": "/.*_ops_.*/",
              "yaxis": 2
            }
          ],
          "spaceLength": 10,
          "stack": false,
          "steppedLine": false,
          "targets": [
            {
              "expr": "[[app]]_metanode_OpMetaReadDir{app=~\"$app\", cluster=\"$cluster\", instance=\"$instance\"}",
              "format": "time_series",
              "hide": false,
              "intervalFactor": 1,
              "legendFormat": "OpMetaReadDir_{{instance}}",
              "refId": "D"
            },
            {
              "expr": "rate([[app]]_metanode_OpMetaReadDir_count{app=~\"$app\", cluster=\"$cluster\", instance=\"$instance\"}[1m])",
              "format": "time_series",
              "hide": false,
              "intervalFactor": 1,
              "legendFormat": "OpMetaReadDir_ops_{{instance}}",
              "refId": "A"
            }
          ],
          "thresholds": [],
          "timeFrom": null,
          "timeShift": null,
          "title": "MetaReadDir",
          "tooltip": {
            "shared": true,
            "sort": 0,
            "value_type": "individual"
          },
          "type": "graph",
          "xaxis": {
            "buckets": null,
            "mode": "time",
            "name": null,
            "show": true,
            "values": []
          },
          "yaxes": [
            {
              "format": "ns",
              "label": null,
              "logBase": 1,
              "max": null,
              "min": null,
              "show": true
            },
            {
              "format": "ops",
              "label": null,
              "logBase": 1,
              "max": null,
              "min": null,
              "show": true
            }
          ],
          "yaxis": {
            "align": false,
            "alignLevel": null
          }
        },
        {
          "aliasColors": {},
          "bars": false,
          "dashLength": 10,
          "dashes": false,
          "datasource": null,
          "fill": 1,
          "gridPos": {
            "h": 8,
            "w": 8,
            "x": 8,
            "y": 35
          },
          "id": 129,
          "legend": {
            "avg": false,
            "current": false,
            "max": false,
            "min": false,
            "show": true,
            "total": false,
            "values": false
          },
          "lines": true,
          "linewidth": 1,
          "links": [],
          "nullPointMode": "null",
          "percentage": false,
          "pointradius": 5,
          "points": false,
          "renderer": "flot",
          "seriesOverrides": [],
          "spaceLength": 10,
          "stack": false,
          "steppedLine": false,
          "targets": [
            {
              "expr": "[[app]]_metanode_OpMetaOpen{app=~\"$app\", cluster=\"$cluster\"}",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "OpMetaOpen_{{instance}}",
              "refId": "A"
            },
            {
              "expr": "[[app]]_metanode_OpMetaLookup{app=~\"$app\", cluster=\"$cluster\"}",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "OpMetaLookup_{{instance}}",
              "refId": "B"
            },
            {
              "expr": "[[app]]_metanode_OpMetaNodeHeartbeat{app=~\"$app\", cluster=\"$cluster\"}",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "OpMetaNodeHeartbeat_{{instance}}",
              "refId": "C"
            },
            {
              "expr": "[[app]]_metanode_OpMetaReadDir{app=~\"$app\", cluster=\"$cluster\"}",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "OpMetaReadDir_{{instance}}",
              "refId": "D"
            },
            {
              "expr": "[[app]]_metanode_OpMetaReleaseOpen{app=~\"$app\", cluster=\"$cluster\"}",
              "format": "time_series",
              "hide": false,
              "intervalFactor": 1,
              "legendFormat": "OpMetaReleaseOpen_{{instance}}",
              "refId": "E"
            },
            {
              "expr": "[[app]]_metanode_OpMetaSetattr{app=~\"$app\", cluster=\"$cluster\"}",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "OpMetaSetattr_{{instance}}",
              "refId": "F"
            },
            {
              "expr": "[[app]]_metanode_OpMetaTruncate{app=~\"$app\", cluster=\"$cluster\"}",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "OpMetaTruncate_{{instance}}",
              "refId": "G"
            }
          ],
          "thresholds": [],
          "timeFrom": null,
          "timeShift": null,
          "title": "MetaReleaseOpen",
          "tooltip": {
            "shared": true,
            "sort": 0,
            "value_type": "individual"
          },
          "type": "graph",
          "xaxis": {
            "buckets": null,
            "mode": "time",
            "name": null,
            "show": true,
            "values": []
          },
          "yaxes": [
            {
              "format": "ns",
              "label": null,
              "logBase": 1,
              "max": null,
              "min": null,
              "show": true
            },
            {
              "format": "short",
              "label": null,
              "logBase": 1,
              "max": null,
              "min": null,
              "show": true
            }
          ],
          "yaxis": {
            "align": false,
            "alignLevel": null
          }
        },
        {
          "aliasColors": {},
          "bars": false,
          "dashLength": 10,
          "dashes": false,
          "datasource": null,
          "fill": 1,
          "gridPos": {
            "h": 8,
            "w": 8,
            "x": 16,
            "y": 35
          },
          "id": 116,
          "legend": {
            "avg": false,
            "current": false,
            "max": false,
            "min": false,
            "show": true,
            "total": false,
            "values": false
          },
          "lines": true,
          "linewidth": 1,
          "links": [],
          "nullPointMode": "null",
          "percentage": false,
          "pointradius": 5,
          "points": false,
          "renderer": "flot",
          "seriesOverrides": [],
          "spaceLength": 10,
          "stack": false,
          "steppedLine": false,
          "targets": [
            {
              "expr": "[[app]]_metanode_OpMetaBatchInodeGet{app=~\"$app\", cluster=\"$cluster\"}",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "OpMetaBatchInodeGet_{{instance}}",
              "refId": "A"
            },
            {
              "expr": "[[app]]_metanode_OpMetaCreateInode{app=~\"$app\", cluster=\"$cluster\"}",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "OpMetaCreateInode",
              "refId": "B"
            },
            {
              "expr": "[[app]]_metanode_OpMetaDeleteInode{app=~\"$app\", cluster=\"$cluster\"}",
              "format": "time_series",
              "hide": false,
              "intervalFactor": 1,
              "legendFormat": "OpMetaDeleteInode_{{instance}}",
              "refId": "C"
            },
            {
              "expr": "[[app]]_metanode_OpMetaEvictInode{app=~\"$app\", cluster=\"$cluster\"}",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "OpMetaEvictInode_{{instance}}",
              "refId": "D"
            },
            {
              "expr": "[[app]]_metanode_OpMetaInodeGet{app=~\"$app\", cluster=\"$cluster\"}",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "OpMetaInodeGet_{{instance}}",
              "refId": "E"
            },
            {
              "expr": "[[app]]_metanode_OpMetaLinkInode{app=~\"$app\", cluster=\"$cluster\"}",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "OpMetaLinkInode_{{instance}}",
              "refId": "F"
            }
          ],
          "thresholds": [],
          "timeFrom": null,
          "timeShift": null,
          "title": "MetaDeleteInode",
          "tooltip": {
            "shared": true,
            "sort": 0,
            "value_type": "individual"
          },
          "type": "graph",
          "xaxis": {
            "buckets": null,
            "mode": "time",
            "name": null,
            "show": true,
            "values": []
          },
          "yaxes": [
            {
              "format": "ns",
              "label": null,
              "logBase": 1,
              "max": null,
              "min": null,
              "show": true
            },
            {
              "format": "short",
              "label": null,
              "logBase": 1,
              "max": null,
              "min": null,
              "show": true
            }
          ],
          "yaxis": {
            "align": false,
            "alignLevel": null
          }
        },
        {
          "aliasColors": {},
          "bars": false,
          "dashLength": 10,
          "dashes": false,
          "datasource": null,
          "fill": 1,
          "gridPos": {
            "h": 8,
            "w": 8,
            "x": 0,
            "y": 43
          },
          "id": 127,
          "legend": {
            "avg": false,
            "current": false,
            "max": false,
            "min": false,
            "show": true,
            "total": false,
            "values": false
          },
          "lines": true,
          "linewidth": 1,
          "links": [],
          "nullPointMode": "null",
          "percentage": false,
          "pointradius": 5,
          "points": false,
          "renderer": "flot",
          "seriesOverrides": [],
          "spaceLength": 10,
          "stack": false,
          "steppedLine": false,
          "targets": [
            {
              "expr": "[[app]]_metanode_OpMetaOpen{app=~\"$app\", cluster=\"$cluster\"}",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "OpMetaOpen_{{instance}}",
              "refId": "A"
            },
            {
              "expr": "[[app]]_metanode_OpMetaLookup{app=~\"$app\", cluster=\"$cluster\"}",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "OpMetaLookup_{{instance}}",
              "refId": "B"
            },
            {
              "expr": "[[app]]_metanode_OpMetaNodeHeartbeat{app=~\"$app\", cluster=\"$cluster\"}",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "OpMetaNodeHeartbeat_{{instance}}",
              "refId": "C"
            },
            {
              "expr": "[[app]]_metanode_OpMetaReadDir{app=~\"$app\", cluster=\"$cluster\"}",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "OpMetaReadDir_{{instance}}",
              "refId": "D"
            },
            {
              "expr": "[[app]]_metanode_OpMetaReleaseOpen{app=~\"$app\", cluster=\"$cluster\"}",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "OpMetaReleaseOpen_{{instance}}",
              "refId": "E"
            },
            {
              "expr": "[[app]]_metanode_OpMetaSetattr{app=~\"$app\", cluster=\"$cluster\"}",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "OpMetaSetattr_{{instance}}",
              "refId": "F"
            },
            {
              "expr": "[[app]]_metanode_OpMetaTruncate{app=~\"$app\", cluster=\"$cluster\"}",
              "format": "time_series",
              "hide": false,
              "intervalFactor": 1,
              "legendFormat": "OpMetaTruncate_{{instance}}",
              "refId": "G"
            }
          ],
          "thresholds": [],
          "timeFrom": null,
          "timeShift": null,
          "title": "MetaTruncate",
          "tooltip": {
            "shared": true,
            "sort": 0,
            "value_type": "individual"
          },
          "type": "graph",
          "xaxis": {
            "buckets": null,
            "mode": "time",
            "name": null,
            "show": true,
            "values": []
          },
          "yaxes": [
            {
              "format": "ns",
              "label": null,
              "logBase": 1,
              "max": null,
              "min": null,
              "show": true
            },
            {
              "format": "short",
              "label": null,
              "logBase": 1,
              "max": null,
              "min": null,
              "show": true
            }
          ],
          "yaxis": {
            "align": false,
            "alignLevel": null
          }
        },
        {
          "aliasColors": {},
          "bars": false,
          "dashLength": 10,
          "dashes": false,
          "datasource": null,
          "fill": 1,
          "gridPos": {
            "h": 8,
            "w": 8,
            "x": 8,
            "y": 43
          },
          "id": 128,
          "legend": {
            "avg": false,
            "current": false,
            "max": false,
            "min": false,
            "show": true,
            "total": false,
            "values": false
          },
          "lines": true,
          "linewidth": 1,
          "links": [],
          "nullPointMode": "null",
          "percentage": false,
          "pointradius": 5,
          "points": false,
          "renderer": "flot",
          "seriesOverrides": [],
          "spaceLength": 10,
          "stack": false,
          "steppedLine": false,
          "targets": [
            {
              "expr": "[[app]]_metanode_OpMetaOpen{app=~\"$app\", cluster=\"$cluster\"}",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "OpMetaOpen_{{instance}}",
              "refId": "A"
            },
            {
              "expr": "[[app]]_metanode_OpMetaLookup{app=~\"$app\", cluster=\"$cluster\"}",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "OpMetaLookup_{{instance}}",
              "refId": "B"
            },
            {
              "expr": "[[app]]_metanode_OpMetaNodeHeartbeat{app=~\"$app\", cluster=\"$cluster\"}",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "OpMetaNodeHeartbeat_{{instance}}",
              "refId": "C"
            },
            {
              "expr": "[[app]]_metanode_OpMetaReadDir{app=~\"$app\", cluster=\"$cluster\"}",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "OpMetaReadDir_{{instance}}",
              "refId": "D"
            },
            {
              "expr": "[[app]]_metanode_OpMetaReleaseOpen{app=~\"$app\", cluster=\"$cluster\"}",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "OpMetaReleaseOpen_{{instance}}",
              "refId": "E"
            },
            {
              "expr": "[[app]]_metanode_OpMetaSetattr{app=~\"$app\", cluster=\"$cluster\"}",
              "format": "time_series",
              "hide": false,
              "intervalFactor": 1,
              "legendFormat": "OpMetaSetattr_{{instance}}",
              "refId": "F"
            },
            {
              "expr": "[[app]]_metanode_OpMetaTruncate{app=~\"$app\", cluster=\"$cluster\"}",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "OpMetaTruncate_{{instance}}",
              "refId": "G"
            }
          ],
          "thresholds": [],
          "timeFrom": null,
          "timeShift": null,
          "title": "MetaSetattr",
          "tooltip": {
            "shared": true,
            "sort": 0,
            "value_type": "individual"
          },
          "type": "graph",
          "xaxis": {
            "buckets": null,
            "mode": "time",
            "name": null,
            "show": true,
            "values": []
          },
          "yaxes": [
            {
              "format": "ns",
              "label": null,
              "logBase": 1,
              "max": null,
              "min": null,
              "show": true
            },
            {
              "format": "short",
              "label": null,
              "logBase": 1,
              "max": null,
              "min": null,
              "show": true
            }
          ],
          "yaxis": {
            "align": false,
            "alignLevel": null
          }
        },
        {
          "aliasColors": {},
          "bars": false,
          "dashLength": 10,
          "dashes": false,
          "datasource": null,
          "fill": 1,
          "gridPos": {
            "h": 8,
            "w": 8,
            "x": 16,
            "y": 43
          },
          "id": 123,
          "legend": {
            "avg": false,
            "current": false,
            "max": false,
            "min": false,
            "show": true,
            "total": false,
            "values": false
          },
          "lines": true,
          "linewidth": 1,
          "links": [],
          "nullPointMode": "null",
          "percentage": false,
          "pointradius": 5,
          "points": false,
          "renderer": "flot",
          "seriesOverrides": [],
          "spaceLength": 10,
          "stack": false,
          "steppedLine": false,
          "targets": [
            {
              "expr": "[[app]]_metanode_OpMetaCreateDentry{app=\"$app\",cluster=~\"$cluster\"}",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "OpMetaCreateDentry_{{instance}}",
              "refId": "A"
            },
            {
              "expr": "[[app]]_metanode_OpMetaDeleteDentry{app=\"$app\",cluster=~\"$cluster\"}",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "OpMetaDeleteDentry_{{instance}}",
              "refId": "B"
            },
            {
              "expr": "[[app]]_metanode_OpMetaUpdateDentry{app=\"$app\",cluster=~\"$cluster\"}",
              "format": "time_series",
              "hide": false,
              "intervalFactor": 1,
              "legendFormat": "OpMetaUpdateDentry_{{instance}}",
              "refId": "C"
            }
          ],
          "thresholds": [],
          "timeFrom": null,
          "timeShift": null,
          "title": "MetaUpdateDentry",
          "tooltip": {
            "shared": true,
            "sort": 0,
            "value_type": "individual"
          },
          "type": "graph",
          "xaxis": {
            "buckets": null,
            "mode": "time",
            "name": null,
            "show": true,
            "values": []
          },
          "yaxes": [
            {
              "format": "ns",
              "label": null,
              "logBase": 1,
              "max": null,
              "min": null,
              "show": true
            },
            {
              "format": "short",
              "label": null,
              "logBase": 1,
              "max": null,
              "min": null,
              "show": true
            }
          ],
          "yaxis": {
            "align": false,
            "alignLevel": null
          }
        },
        {
          "aliasColors": {},
          "bars": false,
          "dashLength": 10,
          "dashes": false,
          "datasource": null,
          "fill": 1,
          "gridPos": {
            "h": 8,
            "w": 8,
            "x": 0,
            "y": 51
          },
          "id": 46,
          "legend": {
            "avg": false,
            "current": false,
            "max": false,
            "min": false,
            "show": true,
            "total": false,
            "values": false
          },
          "lines": true,
          "linewidth": 1,
          "links": [],
          "nullPointMode": "null",
          "percentage": false,
          "pointradius": 5,
          "points": false,
          "renderer": "flot",
          "seriesOverrides": [],
          "spaceLength": 10,
          "stack": false,
          "steppedLine": false,
          "targets": [
            {
              "expr": "[[app]]_metanode_OpMetaOpen{app=~\"$app\", cluster=\"$cluster\"}",
              "format": "time_series",
              "intervalFactor": 1,
              "legendFormat": "OpMetaOpen_{{instance}}",
              "refId": "A"
            },
            {
              "expr": "[[app]]_metanode_OpMetaLookup{app=~\"$app\", cluster=\"$cluster\"}",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "OpMetaLookup_{{instance}}",
              "refId": "B"
            },
            {
              "expr": "[[app]]_metanode_OpMetaNodeHeartbeat{app=~\"$app\", cluster=\"$cluster\"}",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "OpMetaNodeHeartbeat_{{instance}}",
              "refId": "C"
            },
            {
              "expr": "[[app]]_metanode_OpMetaReadDir{app=~\"$app\", cluster=\"$cluster\"}",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "OpMetaReadDir_{{instance}}",
              "refId": "D"
            },
            {
              "expr": "[[app]]_metanode_OpMetaReleaseOpen{app=~\"$app\", cluster=\"$cluster\"}",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "OpMetaReleaseOpen_{{instance}}",
              "refId": "E"
            },
            {
              "expr": "[[app]]_metanode_OpMetaSetattr{app=~\"$app\", cluster=\"$cluster\"}",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "OpMetaSetattr_{{instance}}",
              "refId": "F"
            },
            {
              "expr": "[[app]]_metanode_OpMetaTruncate{app=~\"$app\", cluster=\"$cluster\"}",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "OpMetaTruncate_{{instance}}",
              "refId": "G"
            }
          ],
          "thresholds": [],
          "timeFrom": null,
          "timeShift": null,
          "title": "MetaOpen",
          "tooltip": {
            "shared": true,
            "sort": 0,
            "value_type": "individual"
          },
          "type": "graph",
          "xaxis": {
            "buckets": null,
            "mode": "time",
            "name": null,
            "show": true,
            "values": []
          },
          "yaxes": [
            {
              "format": "ns",
              "label": null,
              "logBase": 1,
              "max": null,
              "min": null,
              "show": true
            },
            {
              "format": "short",
              "label": null,
              "logBase": 1,
              "max": null,
              "min": null,
              "show": true
            }
          ],
          "yaxis": {
            "align": false,
            "alignLevel": null
          }
        }
      ],
      "title": "Metanode",
      "type": "row"
    },
    {
      "collapsed": true,
      "gridPos": {
        "h": 1,
        "w": 24,
        "x": 0,
        "y": 51
      },
      "id": 27,
      "panels": [
        {
          "aliasColors": {},
          "bars": false,
          "dashLength": 10,
          "dashes": false,
          "datasource": null,
          "description": "",
          "fill": 1,
          "gridPos": {
            "h": 8,
            "w": 8,
            "x": 0,
            "y": 52
          },
          "id": 132,
          "legend": {
            "avg": false,
            "current": false,
            "max": false,
            "min": false,
            "show": true,
            "total": false,
            "values": false
          },
          "lines": true,
          "linewidth": 1,
          "links": [],
          "nullPointMode": "null",
          "percentage": false,
          "pointradius": 5,
          "points": false,
          "renderer": "flot",
          "seriesOverrides": [
            {
              "alias": "/.*_ops_.*/",
              "yaxis": 2
            }
          ],
          "spaceLength": 10,
          "stack": false,
          "steppedLine": false,
          "targets": [
            {
              "expr": "[[app]]_dataNode_OpDataNodeHeartbeat{app=~\"$app\", cluster=\"$cluster\"}",
              "format": "time_series",
              "hide": false,
              "intervalFactor": 1,
              "legendFormat": "DataNodeHeartbeat_{{instance}}",
              "refId": "D"
            },
            {
              "expr": "rate([[app]]_dataNode_OpDataNodeHeartbeat{app=~\"$app\", cluster=\"$cluster\"}[1m])",
              "format": "time_series",
              "hide": false,
              "intervalFactor": 1,
              "legendFormat": "DataNodeHeartbeat_ops_{{instance}}",
              "refId": "Q"
            }
          ],
          "thresholds": [],
          "timeFrom": null,
          "timeShift": null,
          "title": "DatanodeHeartbeat",
          "tooltip": {
            "shared": true,
            "sort": 0,
            "value_type": "individual"
          },
          "type": "graph",
          "xaxis": {
            "buckets": null,
            "mode": "time",
            "name": null,
            "show": true,
            "values": []
          },
          "yaxes": [
            {
              "format": "ns",
              "label": null,
              "logBase": 1,
              "max": null,
              "min": null,
              "show": true
            },
            {
              "format": "ops",
              "label": null,
              "logBase": 1,
              "max": null,
              "min": null,
              "show": true
            }
          ],
          "yaxis": {
            "align": false,
            "alignLevel": null
          }
        },
        {
          "aliasColors": {},
          "bars": false,
          "dashLength": 10,
          "dashes": false,
          "datasource": null,
          "fill": 1,
          "gridPos": {
            "h": 8,
            "w": 8,
            "x": 8,
            "y": 52
          },
          "id": 133,
          "legend": {
            "avg": false,
            "current": false,
            "max": false,
            "min": false,
            "show": true,
            "total": false,
            "values": false
          },
          "lines": true,
          "linewidth": 1,
          "links": [],
          "nullPointMode": "null",
          "percentage": false,
          "pointradius": 5,
          "points": false,
          "renderer": "flot",
          "seriesOverrides": [],
          "spaceLength": 10,
          "stack": false,
          "steppedLine": false,
          "targets": [
            {
              "expr": "[[app]]_dataNode_OpBroadcastMinAppliedID{app=~\"$app\", cluster=\"$cluster\"}",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "OpBroadcastMinAppliedID_{{instance}}",
              "refId": "B"
            },
            {
              "expr": "[[app]]_dataNode_OpCreateDataPartition{app=~\"$app\", cluster=\"$cluster\"}",
              "format": "time_series",
              "hide": false,
              "intervalFactor": 1,
              "legendFormat": "OpCreateDataPartition_{{instance}}",
              "refId": "C"
            },
            {
              "expr": "[[app]]_dataNode_OpDataNodeHeartbeat{app=~\"$app\", cluster=\"$cluster\"}",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "DataNodeHeartbeat_{{instance}}",
              "refId": "D"
            },
            {
              "expr": "[[app]]_dataNode_OpDecommissionDataPartition{app=~\"$app\", cluster=\"$cluster\"}",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "DecommissionDataPartition_{{instance}}",
              "refId": "A"
            },
            {
              "expr": "[[app]]_dataNode_OpGetAllWatermarks{app=~\"$app\", cluster=\"$cluster\"}",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "GetAllWatermarks_{{instance}}",
              "refId": "E"
            },
            {
              "expr": "[[app]]_dataNode_OpGetAppliedId{app=~\"$app\", cluster=\"$cluster\"}",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "GetAppliedId_{{instance}}",
              "refId": "F"
            },
            {
              "expr": "[[app]]_dataNode_OpGetPartitionSize{app=~\"$app\", cluster=\"$cluster\"}",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "GetPartitionSize_{{instance}}",
              "refId": "G"
            },
            {
              "expr": "[[app]]_dataNode_OpLoadDataPartition{app=~\"$app\", cluster=\"$cluster\"}",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "LoadDataPartition_{{instance}}",
              "refId": "H"
            },
            {
              "expr": "[[app]]_dataNode_OpMarkDelete{app=~\"$app\", cluster=\"$cluster\"}",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "MarkDelete_{{instance}}",
              "refId": "I"
            },
            {
              "expr": "[[app]]_dataNode_OpNotifyReplicasToRepair{app=~\"$app\", cluster=\"$cluster\"}",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "NotifyReplicasToRepair_{{instance}}",
              "refId": "J"
            },
            {
              "expr": "[[app]]_dataNode_OpRandomWrite{app=~\"$app\", cluster=\"$cluster\"}",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "OpRandomWrite_{{instance}}",
              "refId": "K"
            },
            {
              "expr": "[[app]]_dataNode_OpReadTinyDelete{app=~\"$app\", cluster=\"$cluster\"}",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "ReadTinyDelete_{{instance}}",
              "refId": "L"
            },
            {
              "expr": "[[app]]_dataNode_OpStreamRead{app=~\"$app\", cluster=\"$cluster\"}",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "StreamRead_{{instance}}",
              "refId": "M"
            },
            {
              "expr": "[[app]]_dataNode_OpWrite{app=~\"$app\", cluster=\"$cluster\"}",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "Write_{{instance}}",
              "refId": "N"
            },
            {
              "expr": "[[app]]_dataNode_OpCreateExtent{app=~\"$app\", cluster=\"$cluster\"}",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "CreateExtent_{{instance}}",
              "refId": "O"
            },
            {
              "expr": "[[app]]_dataNode_OpExtentRepairRead{app=~\"$app\", cluster=\"$cluster\"}",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "ExtentRepairdRead_{{instance}}",
              "refId": "P"
            }
          ],
          "thresholds": [],
          "timeFrom": null,
          "timeShift": null,
          "title": "CreateDataPartition",
          "tooltip": {
            "shared": true,
            "sort": 0,
            "value_type": "individual"
          },
          "type": "graph",
          "xaxis": {
            "buckets": null,
            "mode": "time",
            "name": null,
            "show": true,
            "values": []
          },
          "yaxes": [
            {
              "format": "ns",
              "label": null,
              "logBase": 1,
              "max": null,
              "min": null,
              "show": true
            },
            {
              "format": "short",
              "label": null,
              "logBase": 1,
              "max": null,
              "min": null,
              "show": true
            }
          ],
          "yaxis": {
            "align": false,
            "alignLevel": null
          }
        },
        {
          "aliasColors": {},
          "bars": false,
          "dashLength": 10,
          "dashes": false,
          "datasource": null,
          "fill": 1,
          "gridPos": {
            "h": 8,
            "w": 8,
            "x": 16,
            "y": 52
          },
          "id": 137,
          "legend": {
            "avg": false,
            "current": false,
            "max": false,
            "min": false,
            "show": true,
            "total": false,
            "values": false
          },
          "lines": true,
          "linewidth": 1,
          "links": [],
          "nullPointMode": "null",
          "percentage": false,
          "pointradius": 5,
          "points": false,
          "renderer": "flot",
          "seriesOverrides": [
            {
              "alias": "/.*_ops_.*/",
              "yaxis": 2
            }
          ],
          "spaceLength": 10,
          "stack": false,
          "steppedLine": false,
          "targets": [
            {
              "expr": "[[app]]_dataNode_OpLoadDataPartition{app=~\"$app\", cluster=\"$cluster\"}",
              "format": "time_series",
              "hide": false,
              "intervalFactor": 1,
              "legendFormat": "LoadDataPartition_{{instance}}",
              "refId": "H"
            },
            {
              "expr": "rate([[app]]_dataNode_OpLoadDataPartition_count{app=~\"$app\", cluster=\"$cluster\"}[1m])",
              "format": "time_series",
              "hide": false,
              "intervalFactor": 1,
              "legendFormat": "LoadDataPartition_ops_{{instance}}",
              "refId": "A"
            }
          ],
          "thresholds": [],
          "timeFrom": null,
          "timeShift": null,
          "title": "LoadDataPartition",
          "tooltip": {
            "shared": true,
            "sort": 0,
            "value_type": "individual"
          },
          "type": "graph",
          "xaxis": {
            "buckets": null,
            "mode": "time",
            "name": null,
            "show": true,
            "values": []
          },
          "yaxes": [
            {
              "format": "ns",
              "label": null,
              "logBase": 1,
              "max": null,
              "min": null,
              "show": true
            },
            {
              "format": "ops",
              "label": null,
              "logBase": 1,
              "max": null,
              "min": null,
              "show": true
            }
          ],
          "yaxis": {
            "align": false,
            "alignLevel": null
          }
        },
        {
          "aliasColors": {},
          "bars": false,
          "dashLength": 10,
          "dashes": false,
          "datasource": null,
          "fill": 1,
          "gridPos": {
            "h": 8,
            "w": 8,
            "x": 0,
            "y": 60
          },
          "id": 138,
          "legend": {
            "avg": false,
            "current": false,
            "max": false,
            "min": false,
            "show": true,
            "total": false,
            "values": false
          },
          "lines": true,
          "linewidth": 1,
          "links": [],
          "nullPointMode": "null",
          "percentage": false,
          "pointradius": 5,
          "points": false,
          "renderer": "flot",
          "seriesOverrides": [],
          "spaceLength": 10,
          "stack": false,
          "steppedLine": false,
          "targets": [
            {
              "expr": "[[app]]_dataNode_OpBroadcastMinAppliedID{app=~\"$app\", cluster=\"$cluster\"}",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "OpBroadcastMinAppliedID_{{instance}}",
              "refId": "B"
            },
            {
              "expr": "[[app]]_dataNode_OpCreateDataPartition{app=~\"$app\", cluster=\"$cluster\"}",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "CreateDataPartition_{{instance}}",
              "refId": "C"
            },
            {
              "expr": "[[app]]_dataNode_OpDataNodeHeartbeat{app=~\"$app\", cluster=\"$cluster\"}",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "DataNodeHeartbeat_{{instance}}",
              "refId": "D"
            },
            {
              "expr": "[[app]]_dataNode_OpDecommissionDataPartition{app=~\"$app\", cluster=\"$cluster\"}",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "DecommissionDataPartition_{{instance}}",
              "refId": "A"
            },
            {
              "expr": "[[app]]_dataNode_OpGetAllWatermarks{app=~\"$app\", cluster=\"$cluster\"}",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "GetAllWatermarks_{{instance}}",
              "refId": "E"
            },
            {
              "expr": "[[app]]_dataNode_OpGetAppliedId{app=~\"$app\", cluster=\"$cluster\"}",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "GetAppliedId_{{instance}}",
              "refId": "F"
            },
            {
              "expr": "[[app]]_dataNode_OpGetPartitionSize{app=~\"$app\", cluster=\"$cluster\"}",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "GetPartitionSize_{{instance}}",
              "refId": "G"
            },
            {
              "expr": "[[app]]_dataNode_OpLoadDataPartition{app=~\"$app\", cluster=\"$cluster\"}",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "LoadDataPartition_{{instance}}",
              "refId": "H"
            },
            {
              "expr": "[[app]]_dataNode_OpMarkDelete{app=~\"$app\", cluster=\"$cluster\"}",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "MarkDelete_{{instance}}",
              "refId": "I"
            },
            {
              "expr": "[[app]]_dataNode_OpNotifyReplicasToRepair{app=~\"$app\", cluster=\"$cluster\"}",
              "format": "time_series",
              "hide": false,
              "intervalFactor": 1,
              "legendFormat": "NotifyReplicasToRepair_{{instance}}",
              "refId": "J"
            },
            {
              "expr": "[[app]]_dataNode_OpRandomWrite{app=~\"$app\", cluster=\"$cluster\"}",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "OpRandomWrite_{{instance}}",
              "refId": "K"
            },
            {
              "expr": "[[app]]_dataNode_OpReadTinyDelete{app=~\"$app\", cluster=\"$cluster\"}",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "ReadTinyDelete_{{instance}}",
              "refId": "L"
            },
            {
              "expr": "[[app]]_dataNode_OpStreamRead{app=~\"$app\", cluster=\"$cluster\"}",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "StreamRead_{{instance}}",
              "refId": "M"
            },
            {
              "expr": "[[app]]_dataNode_OpWrite{app=~\"$app\", cluster=\"$cluster\"}",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "Write_{{instance}}",
              "refId": "N"
            },
            {
              "expr": "[[app]]_dataNode_OpCreateExtent{app=~\"$app\", cluster=\"$cluster\"}",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "CreateExtent_{{instance}}",
              "refId": "O"
            },
            {
              "expr": "[[app]]_dataNode_OpExtentRepairRead{app=~\"$app\", cluster=\"$cluster\"}",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "ExtentRepairdRead_{{instance}}",
              "refId": "P"
            }
          ],
          "thresholds": [],
          "timeFrom": null,
          "timeShift": null,
          "title": "NotifyReplicasToRepair",
          "tooltip": {
            "shared": true,
            "sort": 0,
            "value_type": "individual"
          },
          "type": "graph",
          "xaxis": {
            "buckets": null,
            "mode": "time",
            "name": null,
            "show": true,
            "values": []
          },
          "yaxes": [
            {
              "format": "ns",
              "label": null,
              "logBase": 1,
              "max": null,
              "min": null,
              "show": true
            },
            {
              "format": "short",
              "label": null,
              "logBase": 1,
              "max": null,
              "min": null,
              "show": true
            }
          ],
          "yaxis": {
            "align": false,
            "alignLevel": null
          }
        },
        {
          "aliasColors": {},
          "bars": false,
          "dashLength": 10,
          "dashes": false,
          "datasource": null,
          "fill": 1,
          "gridPos": {
            "h": 8,
            "w": 8,
            "x": 8,
            "y": 60
          },
          "id": 134,
          "legend": {
            "avg": false,
            "current": false,
            "max": false,
            "min": false,
            "show": true,
            "total": false,
            "values": false
          },
          "lines": true,
          "linewidth": 1,
          "links": [],
          "nullPointMode": "null",
          "percentage": false,
          "pointradius": 5,
          "points": false,
          "renderer": "flot",
          "seriesOverrides": [
            {
              "alias": "/.*_ops_.*/",
              "yaxis": 2
            }
          ],
          "spaceLength": 10,
          "stack": false,
          "steppedLine": false,
          "targets": [
            {
              "expr": "[[app]]_dataNode_OpGetAllWatermarks{app=~\"$app\", cluster=\"$cluster\"}",
              "format": "time_series",
              "hide": false,
              "intervalFactor": 1,
              "legendFormat": "GetAllWatermarks_{{instance}}",
              "refId": "E"
            },
            {
              "expr": "rate([[app]]_dataNode_OpGetAllWatermarks_count{app=~\"$app\", cluster=\"$cluster\"}[1m])",
              "format": "time_series",
              "hide": false,
              "intervalFactor": 1,
              "legendFormat": "GetAllWatermarks_ops_{{instance}}",
              "refId": "A"
            }
          ],
          "thresholds": [],
          "timeFrom": null,
          "timeShift": null,
          "title": "GetAllWatermarks",
          "tooltip": {
            "shared": true,
            "sort": 0,
            "value_type": "individual"
          },
          "type": "graph",
          "xaxis": {
            "buckets": null,
            "mode": "time",
            "name": null,
            "show": true,
            "values": []
          },
          "yaxes": [
            {
              "format": "ns",
              "label": null,
              "logBase": 1,
              "max": null,
              "min": null,
              "show": true
            },
            {
              "format": "ops",
              "label": null,
              "logBase": 1,
              "max": null,
              "min": null,
              "show": true
            }
          ],
          "yaxis": {
            "align": false,
            "alignLevel": null
          }
        },
        {
          "aliasColors": {},
          "bars": false,
          "dashLength": 10,
          "dashes": false,
          "datasource": null,
          "fill": 1,
          "gridPos": {
            "h": 8,
            "w": 8,
            "x": 16,
            "y": 60
          },
          "id": 135,
          "legend": {
            "avg": false,
            "current": false,
            "max": false,
            "min": false,
            "show": true,
            "total": false,
            "values": false
          },
          "lines": true,
          "linewidth": 1,
          "links": [],
          "nullPointMode": "null",
          "percentage": false,
          "pointradius": 5,
          "points": false,
          "renderer": "flot",
          "seriesOverrides": [
            {
              "alias": "/.*_ops_.*/",
              "yaxis": 2
            }
          ],
          "spaceLength": 10,
          "stack": false,
          "steppedLine": false,
          "targets": [
            {
              "expr": "[[app]]_dataNode_OpGetAppliedId{app=~\"$app\", cluster=\"$cluster\"}",
              "format": "time_series",
              "hide": false,
              "intervalFactor": 1,
              "legendFormat": "GetAppliedId_{{instance}}",
              "refId": "F"
            },
            {
              "expr": "rate([[app]]_dataNode_OpGetAppliedId_count{app=~\"$app\", cluster=\"$cluster\"}[1m])",
              "format": "time_series",
              "hide": false,
              "intervalFactor": 1,
              "legendFormat": "GetAppliedId_ops_{{instance}}",
              "refId": "Q"
            }
          ],
          "thresholds": [],
          "timeFrom": null,
          "timeShift": null,
          "title": "GetAppliedId",
          "tooltip": {
            "shared": true,
            "sort": 0,
            "value_type": "individual"
          },
          "type": "graph",
          "xaxis": {
            "buckets": null,
            "mode": "time",
            "name": null,
            "show": true,
            "values": []
          },
          "yaxes": [
            {
              "format": "ns",
              "label": null,
              "logBase": 1,
              "max": null,
              "min": null,
              "show": true
            },
            {
              "format": "ops",
              "label": null,
              "logBase": 1,
              "max": null,
              "min": null,
              "show": true
            }
          ],
          "yaxis": {
            "align": false,
            "alignLevel": null
          }
        },
        {
          "aliasColors": {},
          "bars": false,
          "dashLength": 10,
          "dashes": false,
          "datasource": null,
          "fill": 1,
          "gridPos": {
            "h": 8,
            "w": 8,
            "x": 0,
            "y": 68
          },
          "id": 142,
          "legend": {
            "avg": false,
            "current": false,
            "max": false,
            "min": false,
            "show": true,
            "total": false,
            "values": false
          },
          "lines": true,
          "linewidth": 1,
          "links": [],
          "nullPointMode": "null",
          "percentage": false,
          "pointradius": 5,
          "points": false,
          "renderer": "flot",
          "seriesOverrides": [
            {
              "alias": "/.*_ops_.*/",
              "yaxis": 2
            }
          ],
          "spaceLength": 10,
          "stack": false,
          "steppedLine": false,
          "targets": [
            {
              "expr": "[[app]]_dataNode_OpStreamRead{app=~\"$app\", cluster=\"$cluster\"}",
              "format": "time_series",
              "hide": false,
              "intervalFactor": 1,
              "legendFormat": "StreamRead_{{instance}}",
              "refId": "M"
            },
            {
              "expr": "rate([[app]]dataNode_OpStreamRead_count{app=~\"$app\", cluster=\"$cluster\"}[1m])",
              "format": "time_series",
              "hide": false,
              "intervalFactor": 1,
              "legendFormat": "StreamRead_ops_{{instance}}",
              "refId": "A"
            }
          ],
          "thresholds": [],
          "timeFrom": null,
          "timeShift": null,
          "title": "StreamRead",
          "tooltip": {
            "shared": true,
            "sort": 0,
            "value_type": "individual"
          },
          "type": "graph",
          "xaxis": {
            "buckets": null,
            "mode": "time",
            "name": null,
            "show": true,
            "values": []
          },
          "yaxes": [
            {
              "format": "ns",
              "label": null,
              "logBase": 1,
              "max": null,
              "min": null,
              "show": true
            },
            {
              "format": "ops",
              "label": null,
              "logBase": 1,
              "max": null,
              "min": null,
              "show": true
            }
          ],
          "yaxis": {
            "align": false,
            "alignLevel": null
          }
        },
        {
          "aliasColors": {},
          "bars": false,
          "dashLength": 10,
          "dashes": false,
          "datasource": null,
          "fill": 1,
          "gridPos": {
            "h": 8,
            "w": 8,
            "x": 8,
            "y": 68
          },
          "id": 143,
          "legend": {
            "avg": false,
            "current": false,
            "max": false,
            "min": false,
            "show": true,
            "total": false,
            "values": false
          },
          "lines": true,
          "linewidth": 1,
          "links": [],
          "nullPointMode": "null",
          "percentage": false,
          "pointradius": 5,
          "points": false,
          "renderer": "flot",
          "seriesOverrides": [],
          "spaceLength": 10,
          "stack": false,
          "steppedLine": false,
          "targets": [
            {
              "expr": "[[app]]_dataNode_OpWrite{app=~\"$app\", cluster=\"$cluster\"}",
              "format": "time_series",
              "hide": false,
              "intervalFactor": 1,
              "legendFormat": "Write_{{instance}}",
              "refId": "N"
            },
            {
              "expr": "[[app]]_dataNode_OpCreateExtent{app=~\"$app\", cluster=\"$cluster\"}",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "CreateExtent_{{instance}}",
              "refId": "O"
            },
            {
              "expr": "[[app]]_dataNode_OpExtentRepairRead{app=~\"$app\", cluster=\"$cluster\"}",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "ExtentRepairdRead_{{instance}}",
              "refId": "P"
            }
          ],
          "thresholds": [],
          "timeFrom": null,
          "timeShift": null,
          "title": "Write",
          "tooltip": {
            "shared": true,
            "sort": 0,
            "value_type": "individual"
          },
          "type": "graph",
          "xaxis": {
            "buckets": null,
            "mode": "time",
            "name": null,
            "show": true,
            "values": []
          },
          "yaxes": [
            {
              "format": "ns",
              "label": null,
              "logBase": 1,
              "max": null,
              "min": null,
              "show": true
            },
            {
              "format": "short",
              "label": null,
              "logBase": 1,
              "max": null,
              "min": null,
              "show": true
            }
          ],
          "yaxis": {
            "align": false,
            "alignLevel": null
          }
        },
        {
          "aliasColors": {},
          "bars": false,
          "dashLength": 10,
          "dashes": false,
          "datasource": null,
          "fill": 1,
          "gridPos": {
            "h": 8,
            "w": 8,
            "x": 16,
            "y": 68
          },
          "id": 139,
          "legend": {
            "avg": false,
            "current": false,
            "max": false,
            "min": false,
            "show": true,
            "total": false,
            "values": false
          },
          "lines": true,
          "linewidth": 1,
          "links": [],
          "nullPointMode": "null",
          "percentage": false,
          "pointradius": 5,
          "points": false,
          "renderer": "flot",
          "seriesOverrides": [],
          "spaceLength": 10,
          "stack": false,
          "steppedLine": false,
          "targets": [
            {
              "expr": "[[app]]_dataNode_OpBroadcastMinAppliedID{app=~\"$app\", cluster=\"$cluster\"}",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "OpBroadcastMinAppliedID_{{instance}}",
              "refId": "B"
            },
            {
              "expr": "[[app]]_dataNode_OpCreateDataPartition{app=~\"$app\", cluster=\"$cluster\"}",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "CreateDataPartition_{{instance}}",
              "refId": "C"
            },
            {
              "expr": "[[app]]_dataNode_OpDataNodeHeartbeat{app=~\"$app\", cluster=\"$cluster\"}",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "DataNodeHeartbeat_{{instance}}",
              "refId": "D"
            },
            {
              "expr": "[[app]]_dataNode_OpDecommissionDataPartition{app=~\"$app\", cluster=\"$cluster\"}",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "DecommissionDataPartition_{{instance}}",
              "refId": "A"
            },
            {
              "expr": "[[app]]_dataNode_OpGetAllWatermarks{app=~\"$app\", cluster=\"$cluster\"}",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "GetAllWatermarks_{{instance}}",
              "refId": "E"
            },
            {
              "expr": "[[app]]_dataNode_OpGetAppliedId{app=~\"$app\", cluster=\"$cluster\"}",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "GetAppliedId_{{instance}}",
              "refId": "F"
            },
            {
              "expr": "[[app]]_dataNode_OpGetPartitionSize{app=~\"$app\", cluster=\"$cluster\"}",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "GetPartitionSize_{{instance}}",
              "refId": "G"
            },
            {
              "expr": "[[app]]_dataNode_OpLoadDataPartition{app=~\"$app\", cluster=\"$cluster\"}",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "LoadDataPartition_{{instance}}",
              "refId": "H"
            },
            {
              "expr": "[[app]]_dataNode_OpMarkDelete{app=~\"$app\", cluster=\"$cluster\"}",
              "format": "time_series",
              "hide": false,
              "intervalFactor": 1,
              "legendFormat": "MarkDelete_{{instance}}",
              "refId": "I"
            },
            {
              "expr": "[[app]]_dataNode_OpNotifyReplicasToRepair{app=~\"$app\", cluster=\"$cluster\"}",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "NotifyReplicasToRepair_{{instance}}",
              "refId": "J"
            },
            {
              "expr": "[[app]]_dataNode_OpRandomWrite{app=~\"$app\", cluster=\"$cluster\"}",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "OpRandomWrite_{{instance}}",
              "refId": "K"
            },
            {
              "expr": "[[app]]_dataNode_OpReadTinyDelete{app=~\"$app\", cluster=\"$cluster\"}",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "ReadTinyDelete_{{instance}}",
              "refId": "L"
            },
            {
              "expr": "[[app]]_dataNode_OpStreamRead{app=~\"$app\", cluster=\"$cluster\"}",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "StreamRead_{{instance}}",
              "refId": "M"
            },
            {
              "expr": "[[app]]_dataNode_OpWrite{app=~\"$app\", cluster=\"$cluster\"}",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "Write_{{instance}}",
              "refId": "N"
            },
            {
              "expr": "[[app]]_dataNode_OpCreateExtent{app=~\"$app\", cluster=\"$cluster\"}",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "CreateExtent_{{instance}}",
              "refId": "O"
            },
            {
              "expr": "[[app]]_dataNode_OpExtentRepairRead{app=~\"$app\", cluster=\"$cluster\"}",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "ExtentRepairdRead_{{instance}}",
              "refId": "P"
            }
          ],
          "thresholds": [],
          "timeFrom": null,
          "timeShift": null,
          "title": "MarkDelete",
          "tooltip": {
            "shared": true,
            "sort": 0,
            "value_type": "individual"
          },
          "type": "graph",
          "xaxis": {
            "buckets": null,
            "mode": "time",
            "name": null,
            "show": true,
            "values": []
          },
          "yaxes": [
            {
              "format": "ns",
              "label": null,
              "logBase": 1,
              "max": null,
              "min": null,
              "show": true
            },
            {
              "format": "short",
              "label": null,
              "logBase": 1,
              "max": null,
              "min": null,
              "show": true
            }
          ],
          "yaxis": {
            "align": false,
            "alignLevel": null
          }
        },
        {
          "aliasColors": {},
          "bars": false,
          "dashLength": 10,
          "dashes": false,
          "datasource": null,
          "fill": 1,
          "gridPos": {
            "h": 8,
            "w": 8,
            "x": 0,
            "y": 76
          },
          "id": 144,
          "legend": {
            "avg": false,
            "current": false,
            "max": false,
            "min": false,
            "show": true,
            "total": false,
            "values": false
          },
          "lines": true,
          "linewidth": 1,
          "links": [],
          "nullPointMode": "null",
          "percentage": false,
          "pointradius": 5,
          "points": false,
          "renderer": "flot",
          "seriesOverrides": [
            {
              "alias": "/.*_ops_.*/",
              "yaxis": 2
            }
          ],
          "spaceLength": 10,
          "stack": false,
          "steppedLine": false,
          "targets": [
            {
              "expr": "[[app]]_dataNode_OpCreateExtent{app=~\"$app\", cluster=\"$cluster\"}",
              "format": "time_series",
              "hide": false,
              "intervalFactor": 1,
              "legendFormat": "CreateExtent_{{instance}}",
              "refId": "O"
            },
            {
              "expr": "rate([[app]]_dataNode_OpCreateExtent{app=~\"$app\", cluster=\"$cluster\"}[1m])",
              "format": "time_series",
              "hide": false,
              "intervalFactor": 1,
              "legendFormat": "CreateExtent_ops_{{instance}}",
              "refId": "A"
            }
          ],
          "thresholds": [],
          "timeFrom": null,
          "timeShift": null,
          "title": "CreateExtent",
          "tooltip": {
            "shared": true,
            "sort": 0,
            "value_type": "individual"
          },
          "type": "graph",
          "xaxis": {
            "buckets": null,
            "mode": "time",
            "name": null,
            "show": true,
            "values": []
          },
          "yaxes": [
            {
              "format": "ns",
              "label": null,
              "logBase": 1,
              "max": null,
              "min": null,
              "show": true
            },
            {
              "format": "ops",
              "label": null,
              "logBase": 1,
              "max": null,
              "min": null,
              "show": true
            }
          ],
          "yaxis": {
            "align": false,
            "alignLevel": null
          }
        },
        {
          "aliasColors": {},
          "bars": false,
          "dashLength": 10,
          "dashes": false,
          "datasource": null,
          "fill": 1,
          "gridPos": {
            "h": 8,
            "w": 8,
            "x": 8,
            "y": 76
          },
          "id": 145,
          "legend": {
            "avg": false,
            "current": false,
            "max": false,
            "min": false,
            "show": true,
            "total": false,
            "values": false
          },
          "lines": true,
          "linewidth": 1,
          "links": [],
          "nullPointMode": "null",
          "percentage": false,
          "pointradius": 5,
          "points": false,
          "renderer": "flot",
          "seriesOverrides": [
            {
              "alias": "/.*_ops_.*/",
              "yaxis": 2
            }
          ],
          "spaceLength": 10,
          "stack": false,
          "steppedLine": false,
          "targets": [
            {
              "expr": "[[app]]_dataNode_OpExtentRepairRead{app=~\"$app\", cluster=\"$cluster\"}",
              "format": "time_series",
              "hide": false,
              "intervalFactor": 1,
              "legendFormat": "ExtentRepairdRead_{{instance}}",
              "refId": "P"
            },
            {
              "expr": "rate([[app]]_dataNode_OpExtentRepairRead_count{app=~\"$app\", cluster=\"$cluster\"}[1m])",
              "format": "time_series",
              "hide": false,
              "intervalFactor": 1,
              "legendFormat": "ExtentRepairdRead_ops_{{instance}}",
              "refId": "A"
            }
          ],
          "thresholds": [],
          "timeFrom": null,
          "timeShift": null,
          "title": "ExtentRepairdRead",
          "tooltip": {
            "shared": true,
            "sort": 0,
            "value_type": "individual"
          },
          "type": "graph",
          "xaxis": {
            "buckets": null,
            "mode": "time",
            "name": null,
            "show": true,
            "values": []
          },
          "yaxes": [
            {
              "format": "ns",
              "label": null,
              "logBase": 1,
              "max": null,
              "min": null,
              "show": true
            },
            {
              "format": "short",
              "label": null,
              "logBase": 1,
              "max": null,
              "min": null,
              "show": true
            }
          ],
          "yaxis": {
            "align": false,
            "alignLevel": null
          }
        },
        {
          "aliasColors": {},
          "bars": false,
          "dashLength": 10,
          "dashes": false,
          "datasource": null,
          "fill": 1,
          "gridPos": {
            "h": 8,
            "w": 8,
            "x": 16,
            "y": 76
          },
          "id": 74,
          "legend": {
            "avg": false,
            "current": false,
            "max": false,
            "min": false,
            "show": true,
            "total": false,
            "values": false
          },
          "lines": true,
          "linewidth": 1,
          "links": [],
          "nullPointMode": "null",
          "percentage": false,
          "pointradius": 5,
          "points": false,
          "renderer": "flot",
          "seriesOverrides": [
            {
              "alias": "/.*_ops_.*/",
              "yaxis": 2
            }
          ],
          "spaceLength": 10,
          "stack": false,
          "steppedLine": false,
          "targets": [
            {
              "expr": "[[app]]_dataNode_OpBroadcastMinAppliedID{app=~\"$app\", cluster=\"$cluster\"}",
              "format": "time_series",
              "intervalFactor": 1,
              "legendFormat": "OpBroadcastMinAppliedID_{{instance}}",
              "refId": "B"
            },
            {
              "expr": "rate([[app]]_dataNode_OpBroadcastMinAppliedID{app=~\"$app\", cluster=\"$cluster\"}[1m])",
              "format": "time_series",
              "intervalFactor": 1,
              "legendFormat": "OpBroadcastMinAppliedID_ops_{{instance}}",
              "refId": "Q"
            }
          ],
          "thresholds": [],
          "timeFrom": null,
          "timeShift": null,
          "title": "BroadcastMinAppliedID",
          "tooltip": {
            "shared": true,
            "sort": 0,
            "value_type": "individual"
          },
          "type": "graph",
          "xaxis": {
            "buckets": null,
            "mode": "time",
            "name": null,
            "show": true,
            "values": []
          },
          "yaxes": [
            {
              "format": "ns",
              "label": null,
              "logBase": 1,
              "max": null,
              "min": null,
              "show": true
            },
            {
              "format": "ops",
              "label": null,
              "logBase": 1,
              "max": null,
              "min": null,
              "show": true
            }
          ],
          "yaxis": {
            "align": false,
            "alignLevel": null
          }
        },
        {
          "aliasColors": {},
          "bars": false,
          "dashLength": 10,
          "dashes": false,
          "datasource": null,
          "fill": 1,
          "gridPos": {
            "h": 8,
            "w": 8,
            "x": 0,
            "y": 84
          },
          "id": 141,
          "legend": {
            "avg": false,
            "current": false,
            "max": false,
            "min": false,
            "show": true,
            "total": false,
            "values": false
          },
          "lines": true,
          "linewidth": 1,
          "links": [],
          "nullPointMode": "null",
          "percentage": false,
          "pointradius": 5,
          "points": false,
          "renderer": "flot",
          "seriesOverrides": [
            {
              "alias": "/.*_ops_.*/",
              "yaxis": 2
            }
          ],
          "spaceLength": 10,
          "stack": false,
          "steppedLine": false,
          "targets": [
            {
              "expr": "[[app]]_dataNode_OpReadTinyDelete{app=~\"$app\", cluster=\"$cluster\"}",
              "format": "time_series",
              "hide": false,
              "intervalFactor": 1,
              "legendFormat": "ReadTinyDelete_{{instance}}",
              "refId": "L"
            },
            {
              "expr": "rate([[app]]_dataNode_OpReadTinyDelete_count{app=~\"$app\", cluster=\"$cluster\"}[1m])",
              "format": "time_series",
              "hide": false,
              "intervalFactor": 1,
              "legendFormat": "ReadTinyDelete_ops_{{instance}}",
              "refId": "A"
            }
          ],
          "thresholds": [],
          "timeFrom": null,
          "timeShift": null,
          "title": "ReadTinyDelete",
          "tooltip": {
            "shared": true,
            "sort": 0,
            "value_type": "individual"
          },
          "type": "graph",
          "xaxis": {
            "buckets": null,
            "mode": "time",
            "name": null,
            "show": true,
            "values": []
          },
          "yaxes": [
            {
              "format": "ns",
              "label": null,
              "logBase": 1,
              "max": null,
              "min": null,
              "show": true
            },
            {
              "format": "ops",
              "label": null,
              "logBase": 1,
              "max": null,
              "min": null,
              "show": true
            }
          ],
          "yaxis": {
            "align": false,
            "alignLevel": null
          }
        },
        {
          "aliasColors": {},
          "bars": false,
          "dashLength": 10,
          "dashes": false,
          "datasource": null,
          "fill": 1,
          "gridPos": {
            "h": 8,
            "w": 8,
            "x": 8,
            "y": 84
          },
          "id": 136,
          "legend": {
            "avg": false,
            "current": false,
            "max": false,
            "min": false,
            "show": true,
            "total": false,
            "values": false
          },
          "lines": true,
          "linewidth": 1,
          "links": [],
          "nullPointMode": "null",
          "percentage": false,
          "pointradius": 5,
          "points": false,
          "renderer": "flot",
          "seriesOverrides": [],
          "spaceLength": 10,
          "stack": false,
          "steppedLine": false,
          "targets": [
            {
              "expr": "[[app]]_dataNode_OpBroadcastMinAppliedID{app=~\"$app\", cluster=\"$cluster\"}",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "OpBroadcastMinAppliedID_{{instance}}",
              "refId": "B"
            },
            {
              "expr": "[[app]]_dataNode_OpCreateDataPartition{app=~\"$app\", cluster=\"$cluster\"}",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "CreateDataPartition_{{instance}}",
              "refId": "C"
            },
            {
              "expr": "[[app]]_dataNode_OpDataNodeHeartbeat{app=~\"$app\", cluster=\"$cluster\"}",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "DataNodeHeartbeat_{{instance}}",
              "refId": "D"
            },
            {
              "expr": "[[app]]_dataNode_OpDecommissionDataPartition{app=~\"$app\", cluster=\"$cluster\"}",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "DecommissionDataPartition_{{instance}}",
              "refId": "A"
            },
            {
              "expr": "[[app]]_dataNode_OpGetAllWatermarks{app=~\"$app\", cluster=\"$cluster\"}",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "GetAllWatermarks_{{instance}}",
              "refId": "E"
            },
            {
              "expr": "[[app]]_dataNode_OpGetAppliedId{app=~\"$app\", cluster=\"$cluster\"}",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "GetAppliedId_{{instance}}",
              "refId": "F"
            },
            {
              "expr": "[[app]]_dataNode_OpGetPartitionSize{app=~\"$app\", cluster=\"$cluster\"}",
              "format": "time_series",
              "hide": false,
              "intervalFactor": 1,
              "legendFormat": "GetPartitionSize_{{instance}}",
              "refId": "G"
            },
            {
              "expr": "[[app]]_dataNode_OpLoadDataPartition{app=~\"$app\", cluster=\"$cluster\"}",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "LoadDataPartition_{{instance}}",
              "refId": "H"
            },
            {
              "expr": "[[app]]_dataNode_OpMarkDelete{app=~\"$app\", cluster=\"$cluster\"}",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "MarkDelete_{{instance}}",
              "refId": "I"
            },
            {
              "expr": "[[app]]_dataNode_OpNotifyReplicasToRepair{app=~\"$app\", cluster=\"$cluster\"}",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "NotifyReplicasToRepair_{{instance}}",
              "refId": "J"
            },
            {
              "expr": "[[app]]_dataNode_OpRandomWrite{app=~\"$app\", cluster=\"$cluster\"}",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "OpRandomWrite_{{instance}}",
              "refId": "K"
            },
            {
              "expr": "[[app]]_dataNode_OpReadTinyDelete{app=~\"$app\", cluster=\"$cluster\"}",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "ReadTinyDelete_{{instance}}",
              "refId": "L"
            },
            {
              "expr": "[[app]]_dataNode_OpStreamRead{app=~\"$app\", cluster=\"$cluster\"}",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "StreamRead_{{instance}}",
              "refId": "M"
            },
            {
              "expr": "[[app]]_dataNode_OpWrite{app=~\"$app\", cluster=\"$cluster\"}",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "Write_{{instance}}",
              "refId": "N"
            },
            {
              "expr": "[[app]]_dataNode_OpCreateExtent{app=~\"$app\", cluster=\"$cluster\"}",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "CreateExtent_{{instance}}",
              "refId": "O"
            },
            {
              "expr": "[[app]]_dataNode_OpExtentRepairRead{app=~\"$app\", cluster=\"$cluster\"}",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "ExtentRepairdRead_{{instance}}",
              "refId": "P"
            }
          ],
          "thresholds": [],
          "timeFrom": null,
          "timeShift": null,
          "title": "GetPartitionSize",
          "tooltip": {
            "shared": true,
            "sort": 0,
            "value_type": "individual"
          },
          "type": "graph",
          "xaxis": {
            "buckets": null,
            "mode": "time",
            "name": null,
            "show": true,
            "values": []
          },
          "yaxes": [
            {
              "format": "ns",
              "label": null,
              "logBase": 1,
              "max": null,
              "min": null,
              "show": true
            },
            {
              "format": "short",
              "label": null,
              "logBase": 1,
              "max": null,
              "min": null,
              "show": true
            }
          ],
          "yaxis": {
            "align": false,
            "alignLevel": null
          }
        },
        {
          "aliasColors": {},
          "bars": false,
          "dashLength": 10,
          "dashes": false,
          "datasource": null,
          "fill": 1,
          "gridPos": {
            "h": 8,
            "w": 8,
            "x": 16,
            "y": 84
          },
          "id": 146,
          "legend": {
            "avg": false,
            "current": false,
            "max": false,
            "min": false,
            "show": true,
            "total": false,
            "values": false
          },
          "lines": true,
          "linewidth": 1,
          "links": [],
          "nullPointMode": "null",
          "percentage": false,
          "pointradius": 5,
          "points": false,
          "renderer": "flot",
          "seriesOverrides": [
            {
              "alias": "/.*_ops_.*/",
              "yaxis": 2
            }
          ],
          "spaceLength": 10,
          "stack": false,
          "steppedLine": false,
          "targets": [
            {
              "expr": "[[app]]_dataNode_OpTinyExtentRepairRead{app=~\"$app\", cluster=\"$cluster\"}",
              "format": "time_series",
              "hide": false,
              "intervalFactor": 1,
              "legendFormat": "TinyExtentRepairdRead_{{instance}}",
              "refId": "P"
            },
            {
              "expr": "rate([[app]]_dataNode_OpTinyExtentRepairRead_count{app=~\"$app\", cluster=\"$cluster\"}[1m])",
              "format": "time_series",
              "hide": false,
              "intervalFactor": 1,
              "legendFormat": "TinyExtentRepairdRead_ops_{{instance}}",
              "refId": "A"
            }
          ],
          "thresholds": [],
          "timeFrom": null,
          "timeShift": null,
          "title": "TinyExtentRepairdRead",
          "tooltip": {
            "shared": true,
            "sort": 0,
            "value_type": "individual"
          },
          "type": "graph",
          "xaxis": {
            "buckets": null,
            "mode": "time",
            "name": null,
            "show": true,
            "values": []
          },
          "yaxes": [
            {
              "format": "ns",
              "label": null,
              "logBase": 1,
              "max": null,
              "min": null,
              "show": true
            },
            {
              "format": "ops",
              "label": null,
              "logBase": 1,
              "max": null,
              "min": null,
              "show": true
            }
          ],
          "yaxis": {
            "align": false,
            "alignLevel": null
          }
        },
        {
          "aliasColors": {},
          "bars": false,
          "dashLength": 10,
          "dashes": false,
          "datasource": null,
          "fill": 1,
          "gridPos": {
            "h": 8,
            "w": 8,
            "x": 0,
            "y": 92
          },
          "id": 131,
          "legend": {
            "avg": false,
            "current": false,
            "max": false,
            "min": false,
            "show": true,
            "total": false,
            "values": false
          },
          "lines": true,
          "linewidth": 1,
          "links": [],
          "nullPointMode": "null",
          "percentage": false,
          "pointradius": 5,
          "points": false,
          "renderer": "flot",
          "seriesOverrides": [],
          "spaceLength": 10,
          "stack": false,
          "steppedLine": false,
          "targets": [
            {
              "expr": "[[app]]_dataNode_OpBroadcastMinAppliedID{app=~\"$app\", cluster=\"$cluster\"}",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "OpBroadcastMinAppliedID_{{instance}}",
              "refId": "B"
            },
            {
              "expr": "[[app]]_dataNode_OpCreateDataPartition{app=~\"$app\", cluster=\"$cluster\"}",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "CreateDataPartition_{{instance}}",
              "refId": "C"
            },
            {
              "expr": "[[app]]_dataNode_OpDataNodeHeartbeat{app=~\"$app\", cluster=\"$cluster\"}",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "DataNodeHeartbeat_{{instance}}",
              "refId": "D"
            },
            {
              "expr": "[[app]]_dataNode_OpDecommissionDataPartition{app=~\"$app\", cluster=\"$cluster\"}",
              "format": "time_series",
              "hide": false,
              "intervalFactor": 1,
              "legendFormat": "DecommissionDataPartition_{{instance}}",
              "refId": "A"
            },
            {
              "expr": "[[app]]_dataNode_OpGetAllWatermarks{app=~\"$app\", cluster=\"$cluster\"}",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "GetAllWatermarks_{{instance}}",
              "refId": "E"
            },
            {
              "expr": "[[app]]_dataNode_OpGetAppliedId{app=~\"$app\", cluster=\"$cluster\"}",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "GetAppliedId_{{instance}}",
              "refId": "F"
            },
            {
              "expr": "[[app]]_dataNode_OpGetPartitionSize{app=~\"$app\", cluster=\"$cluster\"}",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "GetPartitionSize_{{instance}}",
              "refId": "G"
            },
            {
              "expr": "[[app]]_dataNode_OpLoadDataPartition{app=~\"$app\", cluster=\"$cluster\"}",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "LoadDataPartition_{{instance}}",
              "refId": "H"
            },
            {
              "expr": "[[app]]_dataNode_OpMarkDelete{app=~\"$app\", cluster=\"$cluster\"}",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "MarkDelete_{{instance}}",
              "refId": "I"
            },
            {
              "expr": "[[app]]_dataNode_OpNotifyReplicasToRepair{app=~\"$app\", cluster=\"$cluster\"}",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "NotifyReplicasToRepair_{{instance}}",
              "refId": "J"
            },
            {
              "expr": "[[app]]_dataNode_OpRandomWrite{app=~\"$app\", cluster=\"$cluster\"}",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "OpRandomWrite_{{instance}}",
              "refId": "K"
            },
            {
              "expr": "[[app]]_dataNode_OpReadTinyDelete{app=~\"$app\", cluster=\"$cluster\"}",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "ReadTinyDelete_{{instance}}",
              "refId": "L"
            },
            {
              "expr": "[[app]]_dataNode_OpStreamRead{app=~\"$app\", cluster=\"$cluster\"}",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "StreamRead_{{instance}}",
              "refId": "M"
            },
            {
              "expr": "[[app]]_dataNode_OpWrite{app=~\"$app\", cluster=\"$cluster\"}",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "Write_{{instance}}",
              "refId": "N"
            },
            {
              "expr": "[[app]]_dataNode_OpCreateExtent{app=~\"$app\", cluster=\"$cluster\"}",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "CreateExtent_{{instance}}",
              "refId": "O"
            },
            {
              "expr": "[[app]]_dataNode_OpExtentRepairRead{app=~\"$app\", cluster=\"$cluster\"}",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "ExtentRepairdRead_{{instance}}",
              "refId": "P"
            }
          ],
          "thresholds": [],
          "timeFrom": null,
          "timeShift": null,
          "title": "DecommissionDataPartition",
          "tooltip": {
            "shared": true,
            "sort": 0,
            "value_type": "individual"
          },
          "type": "graph",
          "xaxis": {
            "buckets": null,
            "mode": "time",
            "name": null,
            "show": true,
            "values": []
          },
          "yaxes": [
            {
              "format": "ns",
              "label": null,
              "logBase": 1,
              "max": null,
              "min": null,
              "show": true
            },
            {
              "format": "short",
              "label": null,
              "logBase": 1,
              "max": null,
              "min": null,
              "show": true
            }
          ],
          "yaxis": {
            "align": false,
            "alignLevel": null
          }
        },
        {
          "aliasColors": {},
          "bars": false,
          "dashLength": 10,
          "dashes": false,
          "datasource": null,
          "fill": 1,
          "gridPos": {
            "h": 8,
            "w": 8,
            "x": 8,
            "y": 92
          },
          "id": 140,
          "legend": {
            "avg": false,
            "current": false,
            "max": false,
            "min": false,
            "show": true,
            "total": false,
            "values": false
          },
          "lines": true,
          "linewidth": 1,
          "links": [],
          "nullPointMode": "null",
          "percentage": false,
          "pointradius": 5,
          "points": false,
          "renderer": "flot",
          "seriesOverrides": [],
          "spaceLength": 10,
          "stack": false,
          "steppedLine": false,
          "targets": [
            {
              "expr": "[[app]]_dataNode_OpBroadcastMinAppliedID{app=~\"$app\", cluster=\"$cluster\"}",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "OpBroadcastMinAppliedID_{{instance}}",
              "refId": "B"
            },
            {
              "expr": "[[app]]_dataNode_OpCreateDataPartition{app=~\"$app\", cluster=\"$cluster\"}",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "CreateDataPartition_{{instance}}",
              "refId": "C"
            },
            {
              "expr": "[[app]]_dataNode_OpDataNodeHeartbeat{app=~\"$app\", cluster=\"$cluster\"}",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "DataNodeHeartbeat_{{instance}}",
              "refId": "D"
            },
            {
              "expr": "[[app]]_dataNode_OpDecommissionDataPartition{app=~\"$app\", cluster=\"$cluster\"}",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "DecommissionDataPartition_{{instance}}",
              "refId": "A"
            },
            {
              "expr": "[[app]]_dataNode_OpGetAllWatermarks{app=~\"$app\", cluster=\"$cluster\"}",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "GetAllWatermarks_{{instance}}",
              "refId": "E"
            },
            {
              "expr": "[[app]]_dataNode_OpGetAppliedId{app=~\"$app\", cluster=\"$cluster\"}",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "GetAppliedId_{{instance}}",
              "refId": "F"
            },
            {
              "expr": "[[app]]_dataNode_OpGetPartitionSize{app=~\"$app\", cluster=\"$cluster\"}",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "GetPartitionSize_{{instance}}",
              "refId": "G"
            },
            {
              "expr": "[[app]]_dataNode_OpLoadDataPartition{app=~\"$app\", cluster=\"$cluster\"}",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "LoadDataPartition_{{instance}}",
              "refId": "H"
            },
            {
              "expr": "[[app]]_dataNode_OpMarkDelete{app=~\"$app\", cluster=\"$cluster\"}",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "MarkDelete_{{instance}}",
              "refId": "I"
            },
            {
              "expr": "[[app]]_dataNode_OpNotifyReplicasToRepair{app=~\"$app\", cluster=\"$cluster\"}",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "NotifyReplicasToRepair_{{instance}}",
              "refId": "J"
            },
            {
              "expr": "[[app]]_dataNode_OpRandomWrite{app=~\"$app\", cluster=\"$cluster\"}",
              "format": "time_series",
              "hide": false,
              "intervalFactor": 1,
              "legendFormat": "OpRandomWrite_{{instance}}",
              "refId": "K"
            },
            {
              "expr": "[[app]]_dataNode_OpReadTinyDelete{app=~\"$app\", cluster=\"$cluster\"}",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "ReadTinyDelete_{{instance}}",
              "refId": "L"
            },
            {
              "expr": "[[app]]_dataNode_OpStreamRead{app=~\"$app\", cluster=\"$cluster\"}",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "StreamRead_{{instance}}",
              "refId": "M"
            },
            {
              "expr": "[[app]]_dataNode_OpWrite{app=~\"$app\", cluster=\"$cluster\"}",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "Write_{{instance}}",
              "refId": "N"
            },
            {
              "expr": "[[app]]_dataNode_OpCreateExtent{app=~\"$app\", cluster=\"$cluster\"}",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "CreateExtent_{{instance}}",
              "refId": "O"
            },
            {
              "expr": "[[app]]_dataNode_OpExtentRepairRead{app=~\"$app\", cluster=\"$cluster\"}",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "ExtentRepairdRead_{{instance}}",
              "refId": "P"
            }
          ],
          "thresholds": [],
          "timeFrom": null,
          "timeShift": null,
          "title": "RandomWrite",
          "tooltip": {
            "shared": true,
            "sort": 0,
            "value_type": "individual"
          },
          "type": "graph",
          "xaxis": {
            "buckets": null,
            "mode": "time",
            "name": null,
            "show": true,
            "values": []
          },
          "yaxes": [
            {
              "format": "ns",
              "label": null,
              "logBase": 1,
              "max": null,
              "min": null,
              "show": true
            },
            {
              "format": "short",
              "label": null,
              "logBase": 1,
              "max": null,
              "min": null,
              "show": true
            }
          ],
          "yaxis": {
            "align": false,
            "alignLevel": null
          }
        }
      ],
      "title": "Datanode",
      "type": "row"
    },
    {
      "collapsed": true,
      "gridPos": {
        "h": 1,
        "w": 24,
        "x": 0,
        "y": 52
      },
      "id": 66,
      "panels": [
        {
          "aliasColors": {},
          "bars": false,
          "dashLength": 10,
          "dashes": false,
          "datasource": null,
          "fill": 1,
          "gridPos": {
            "h": 7,
            "w": 8,
            "x": 0,
            "y": 5
          },
          "id": 64,
          "legend": {
            "avg": false,
            "current": false,
            "max": false,
            "min": false,
            "show": true,
            "total": false,
            "values": false
          },
          "lines": true,
          "linewidth": 1,
          "links": [],
          "nullPointMode": "null",
          "percentage": false,
          "pointradius": 5,
          "points": false,
          "renderer": "flot",
          "seriesOverrides": [
            {
              "alias": "/.*_ops_.*/",
              "yaxis": 2
            }
          ],
          "spaceLength": 10,
          "stack": false,
          "steppedLine": false,
          "targets": [
            {
              "expr": "[[app]]_fuseclient_OpMetaReadDir{app=\"$app\", cluster=\"$cluster\", instance=\"$instance\"}",
              "format": "time_series",
              "hide": false,
              "intervalFactor": 1,
              "legendFormat": "MetaReaddir_{{instance}}",
              "refId": "K"
            },
            {
              "expr": "rate([[app]]_fuseclient_OpMetaReadDir_count{app=\"$app\", cluster=\"$cluster\", instance=\"$instance\"}[1m])",
              "format": "time_series",
              "hide": false,
              "intervalFactor": 1,
              "legendFormat": "MetaReaddir_ops_{{instance}}",
              "refId": "E"
            }
          ],
          "thresholds": [],
          "timeFrom": null,
          "timeShift": null,
          "title": "MetaReadDir",
          "tooltip": {
            "shared": true,
            "sort": 0,
            "value_type": "individual"
          },
          "type": "graph",
          "xaxis": {
            "buckets": null,
            "mode": "time",
            "name": null,
            "show": true,
            "values": []
          },
          "yaxes": [
            {
              "format": "ns",
              "label": null,
              "logBase": 1,
              "max": null,
              "min": null,
              "show": true
            },
            {
              "format": "ops",
              "label": null,
              "logBase": 1,
              "max": null,
              "min": null,
              "show": true
            }
          ],
          "yaxis": {
            "align": false,
            "alignLevel": null
          }
        },
        {
          "aliasColors": {},
          "bars": false,
          "dashLength": 10,
          "dashes": false,
          "datasource": null,
          "fill": 1,
          "gridPos": {
            "h": 7,
            "w": 8,
            "x": 8,
            "y": 5
          },
          "id": 157,
          "legend": {
            "avg": false,
            "current": false,
            "max": false,
            "min": false,
            "show": true,
            "total": false,
            "values": false
          },
          "lines": true,
          "linewidth": 1,
          "links": [],
          "nullPointMode": "null",
          "percentage": false,
          "pointradius": 5,
          "points": false,
          "renderer": "flot",
          "seriesOverrides": [
            {
              "alias": "/.*_count_.*/",
              "yaxis": 2
            }
          ],
          "spaceLength": 10,
          "stack": false,
          "steppedLine": false,
          "targets": [
            {
              "expr": "[[app]]_fuseclient_OpMetaBatchInodeGet{app=~\"$app\", cluster=\"$cluster\"}",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "OpMetaBatchInodeGet_{{instance}}",
              "refId": "A"
            },
            {
              "expr": "[[app]]_fuseclient_OpMetaCreateInode{app=~\"$app\", cluster=\"$cluster\", instance=\"$instance\"}",
              "format": "time_series",
              "hide": false,
              "intervalFactor": 1,
              "legendFormat": "CreateInode_{{instance}}",
              "refId": "D"
            },
            {
              "expr": "[[app]]_fuseclient_OpMetaDeleteInode{app=~\"$app\", cluster=\"$cluster\"}",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "DeleteInode_{{instance}}",
              "refId": "F"
            },
            {
              "expr": "[[app]]_fuseclient_OpMetaEvictInode{app=~\"$app\", cluster=\"$cluster\"}",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "EvictInode_{{instance}}",
              "refId": "G"
            },
            {
              "expr": "[[app]]_fuseclient_OpMetaInodeGet{app=~\"$app\", cluster=\"$cluster\"}",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "InodeGet_{{instance}}",
              "refId": "J"
            },
            {
              "expr": "[[app]]_fuseclient_OpMetaTruncate{app=~\"$app\", cluster=\"$cluster\"}",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "Truncate_{{instance}}",
              "refId": "B"
            },
            {
              "expr": "rate([[app]]_fuseclient_OpMetaBatchInodeGet_count{app=~\"$app\", cluster=\"$cluster\"}[1m])",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "OpMetaBatchInodeGet_count_{{instance}}",
              "refId": "C"
            },
            {
              "expr": "rate([[app]]_fuseclient_OpMetaCreateInode{app=~\"$app\", cluster=\"$cluster\", instance=\"$instance\"}[1m])",
              "format": "time_series",
              "hide": false,
              "intervalFactor": 1,
              "legendFormat": "CreateInode_count_{{instance}}",
              "refId": "E"
            },
            {
              "expr": "rate([[app]]_fuseclient_OpMetaDeleteInode{app=~\"$app\", cluster=\"$cluster\"}[1m])",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "DeleteInode_count_{{instance}}",
              "refId": "H"
            },
            {
              "expr": "rate([[app]]_fuseclient_OpMetaEvictInode{app=~\"$app\", cluster=\"$cluster\"}[1m])",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "EvictInode_count_{{instance}}",
              "refId": "I"
            },
            {
              "expr": "rate([[app]]_fuseclient_OpMetaInodeGet{app=~\"$app\", cluster=\"$cluster\"}[1m])",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "InodeGet_count_{{instance}}",
              "refId": "K"
            },
            {
              "expr": "rate([[app]]_fuseclient_OpMetaTruncate{app=~\"$app\", cluster=\"$cluster\"}[1m])",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "Truncate_count_{{instance}}",
              "refId": "L"
            }
          ],
          "thresholds": [],
          "timeFrom": null,
          "timeShift": null,
          "title": "MetaCreateInode",
          "tooltip": {
            "shared": true,
            "sort": 0,
            "value_type": "individual"
          },
          "type": "graph",
          "xaxis": {
            "buckets": null,
            "mode": "time",
            "name": null,
            "show": true,
            "values": []
          },
          "yaxes": [
            {
              "format": "ns",
              "label": null,
              "logBase": 1,
              "max": null,
              "min": null,
              "show": true
            },
            {
              "format": "ops",
              "label": null,
              "logBase": 1,
              "max": null,
              "min": null,
              "show": true
            }
          ],
          "yaxis": {
            "align": false,
            "alignLevel": null
          }
        },
        {
          "aliasColors": {},
          "bars": false,
          "dashLength": 10,
          "dashes": false,
          "datasource": null,
          "fill": 1,
          "gridPos": {
            "h": 7,
            "w": 8,
            "x": 16,
            "y": 5
          },
          "id": 156,
          "legend": {
            "avg": false,
            "current": false,
            "max": false,
            "min": false,
            "show": true,
            "total": false,
            "values": false
          },
          "lines": true,
          "linewidth": 1,
          "links": [],
          "nullPointMode": "null",
          "percentage": false,
          "pointradius": 5,
          "points": false,
          "renderer": "flot",
          "seriesOverrides": [
            {
              "alias": "/.*_count_.*/",
              "yaxis": 2
            }
          ],
          "spaceLength": 10,
          "stack": false,
          "steppedLine": false,
          "targets": [
            {
              "expr": "[[app]]_fuseclient_OpMetaBatchInodeGet{app=~\"$app\", cluster=\"$cluster\"}",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "OpMetaBatchInodeGet_{{instance}}",
              "refId": "A"
            },
            {
              "expr": "[[app]]_fuseclient_OpMetaCreateInode{app=~\"$app\", cluster=\"$cluster\"}",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "CreateInode_{{instance}}",
              "refId": "D"
            },
            {
              "expr": "[[app]]_fuseclient_OpMetaDeleteInode{app=~\"$app\", cluster=\"$cluster\", instance=\"$instance\"}",
              "format": "time_series",
              "hide": false,
              "intervalFactor": 1,
              "legendFormat": "DeleteInode_{{instance}}",
              "refId": "F"
            },
            {
              "expr": "[[app]]_fuseclient_OpMetaEvictInode{app=~\"$app\", cluster=\"$cluster\"}",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "EvictInode_{{instance}}",
              "refId": "G"
            },
            {
              "expr": "[[app]]_fuseclient_OpMetaInodeGet{app=~\"$app\", cluster=\"$cluster\"}",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "InodeGet_{{instance}}",
              "refId": "J"
            },
            {
              "expr": "[[app]]_fuseclient_OpMetaTruncate{app=~\"$app\", cluster=\"$cluster\"}",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "Truncate_{{instance}}",
              "refId": "B"
            },
            {
              "expr": "rate([[app]]_fuseclient_OpMetaBatchInodeGet_count{app=~\"$app\", cluster=\"$cluster\", instance=\"$instance\"}[1m])",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "OpMetaBatchInodeGet_count_{{instance}}",
              "refId": "C"
            },
            {
              "expr": "rate([[app]]_fuseclient_OpMetaCreateInode{app=~\"$app\", cluster=\"$cluster\"}[1m])",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "CreateInode_count_{{instance}}",
              "refId": "E"
            },
            {
              "expr": "rate([[app]]_fuseclient_OpMetaDeleteInode_count{app=~\"$app\", cluster=\"$cluster\", instance=\"$instance\"}[1m])",
              "format": "time_series",
              "hide": false,
              "intervalFactor": 1,
              "legendFormat": "DeleteInode_count_{{instance}}",
              "refId": "H"
            },
            {
              "expr": "rate([[app]]_fuseclient_OpMetaEvictInode{app=~\"$app\", cluster=\"$cluster\"}[1m])",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "EvictInode_count_{{instance}}",
              "refId": "I"
            },
            {
              "expr": "rate([[app]]_fuseclient_OpMetaInodeGet{app=~\"$app\", cluster=\"$cluster\"}[1m])",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "InodeGet_count_{{instance}}",
              "refId": "K"
            },
            {
              "expr": "rate([[app]]_fuseclient_OpMetaTruncate{app=~\"$app\", cluster=\"$cluster\"}[1m])",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "Truncate_count_{{instance}}",
              "refId": "L"
            }
          ],
          "thresholds": [],
          "timeFrom": null,
          "timeShift": null,
          "title": "DeleteInode",
          "tooltip": {
            "shared": true,
            "sort": 0,
            "value_type": "individual"
          },
          "type": "graph",
          "xaxis": {
            "buckets": null,
            "mode": "time",
            "name": null,
            "show": true,
            "values": []
          },
          "yaxes": [
            {
              "format": "ns",
              "label": null,
              "logBase": 1,
              "max": null,
              "min": null,
              "show": true
            },
            {
              "format": "ops",
              "label": null,
              "logBase": 1,
              "max": null,
              "min": null,
              "show": true
            }
          ],
          "yaxis": {
            "align": false,
            "alignLevel": null
          }
        },
        {
          "aliasColors": {},
          "bars": false,
          "dashLength": 10,
          "dashes": false,
          "datasource": null,
          "fill": 1,
          "gridPos": {
            "h": 7,
            "w": 8,
            "x": 0,
            "y": 12
          },
          "id": 160,
          "legend": {
            "avg": false,
            "current": false,
            "max": false,
            "min": false,
            "show": true,
            "total": false,
            "values": false
          },
          "lines": true,
          "linewidth": 1,
          "links": [],
          "nullPointMode": "null",
          "percentage": false,
          "pointradius": 5,
          "points": false,
          "renderer": "flot",
          "seriesOverrides": [
            {
              "alias": "/.*_count_.*/",
              "yaxis": 2
            }
          ],
          "spaceLength": 10,
          "stack": false,
          "steppedLine": false,
          "targets": [
            {
              "expr": "[[app]]_fuseclient_OpMetaCreateDentry{app=~\"$app\", cluster=\"$cluster\"}",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "OpMetaCreateDentry_{{instance}}",
              "refId": "C"
            },
            {
              "expr": "[[app]]_fuseclient_OpMetaDeleteDentry{app=~\"$app\", cluster=\"$cluster\", instance=\"$instance\"}",
              "format": "time_series",
              "hide": false,
              "intervalFactor": 1,
              "legendFormat": "OpMetaDeleteDentry_{{instance}}",
              "refId": "E"
            },
            {
              "expr": "[[app]]_fuseclient_OpMetaUpdateDentry{app=~\"$app\", cluster=\"$cluster\"}",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "OpMetaUpdateDentry_{{instance}}",
              "refId": "A"
            },
            {
              "expr": "[[app]]_fuseclient_OpMetaCreateDentry_count{app=~\"$app\", cluster=\"$cluster\"}",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "OpMetaCreateDentry_count_{{instance}}",
              "refId": "B"
            },
            {
              "expr": "rate([[app]]_fuseclient_OpMetaDeleteDentry_count{app=~\"$app\", cluster=\"$cluster\", instance=\"$instance\"}[1m])",
              "format": "time_series",
              "hide": false,
              "intervalFactor": 1,
              "legendFormat": "OpMetaDeleteDentry_count_{{instance}}",
              "refId": "D"
            },
            {
              "expr": "[[app]]_fuseclient_OpMetaUpdateDentry_count{app=~\"$app\", cluster=\"$cluster\"}",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "OpMetaUpdateDentry_count_{{instance}}",
              "refId": "F"
            }
          ],
          "thresholds": [],
          "timeFrom": null,
          "timeShift": null,
          "title": "MetaDeleteDentry",
          "tooltip": {
            "shared": true,
            "sort": 0,
            "value_type": "individual"
          },
          "type": "graph",
          "xaxis": {
            "buckets": null,
            "mode": "time",
            "name": null,
            "show": true,
            "values": []
          },
          "yaxes": [
            {
              "format": "ns",
              "label": null,
              "logBase": 1,
              "max": null,
              "min": null,
              "show": true
            },
            {
              "format": "ops",
              "label": null,
              "logBase": 1,
              "max": null,
              "min": null,
              "show": true
            }
          ],
          "yaxis": {
            "align": false,
            "alignLevel": null
          }
        },
        {
          "aliasColors": {},
          "bars": false,
          "dashLength": 10,
          "dashes": false,
          "datasource": null,
          "fill": 1,
          "gridPos": {
            "h": 7,
            "w": 8,
            "x": 8,
            "y": 12
          },
          "id": 68,
          "legend": {
            "avg": false,
            "current": false,
            "max": false,
            "min": false,
            "show": true,
            "total": false,
            "values": false
          },
          "lines": true,
          "linewidth": 1,
          "links": [],
          "nullPointMode": "null",
          "percentage": false,
          "pointradius": 5,
          "points": false,
          "renderer": "flot",
          "seriesOverrides": [
            {
              "alias": "/.*_count_.*/",
              "yaxis": 2
            }
          ],
          "spaceLength": 10,
          "stack": false,
          "steppedLine": false,
          "targets": [
            {
              "expr": "[[app]]_fuseclient_OpMetaCreateDentry{app=~\"$app\", cluster=\"$cluster\", instance=\"$instance\"}",
              "format": "time_series",
              "intervalFactor": 1,
              "legendFormat": "OpMetaCreateDentry_{{instance}}",
              "refId": "C"
            },
            {
              "expr": "[[app]]_fuseclient_OpMetaDeleteDentry{app=~\"$app\", cluster=\"$cluster\"}",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "OpMetaDeleteDentry_{{instance}}",
              "refId": "E"
            },
            {
              "expr": "[[app]]_fuseclient_OpMetaUpdateDentry{app=~\"$app\", cluster=\"$cluster\"}",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "OpMetaUpdateDentry_{{instance}}",
              "refId": "A"
            },
            {
              "expr": "rate([[app]]_fuseclient_OpMetaCreateDentry_count{app=~\"$app\", cluster=\"$cluster\", instance=\"$instance\"}[1m])",
              "format": "time_series",
              "hide": false,
              "intervalFactor": 1,
              "legendFormat": "OpMetaCreateDentry_count_{{instance}}",
              "refId": "B"
            },
            {
              "expr": "[[app]]_fuseclient_OpMetaDeleteDentry_count{app=~\"$app\", cluster=\"$cluster\"}",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "OpMetaDeleteDentry_count_{{instance}}",
              "refId": "D"
            },
            {
              "expr": "[[app]]_fuseclient_OpMetaUpdateDentry_count{app=~\"$app\", cluster=\"$cluster\"}",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "OpMetaUpdateDentry_count_{{instance}}",
              "refId": "F"
            }
          ],
          "thresholds": [],
          "timeFrom": null,
          "timeShift": null,
          "title": "MetaCreateDentry",
          "tooltip": {
            "shared": true,
            "sort": 0,
            "value_type": "individual"
          },
          "type": "graph",
          "xaxis": {
            "buckets": null,
            "mode": "time",
            "name": null,
            "show": true,
            "values": []
          },
          "yaxes": [
            {
              "format": "ns",
              "label": null,
              "logBase": 1,
              "max": null,
              "min": null,
              "show": true
            },
            {
              "format": "ops",
              "label": null,
              "logBase": 1,
              "max": null,
              "min": null,
              "show": true
            }
          ],
          "yaxis": {
            "align": false,
            "alignLevel": null
          }
        },
        {
          "aliasColors": {},
          "bars": false,
          "dashLength": 10,
          "dashes": false,
          "datasource": null,
          "fill": 1,
          "gridPos": {
            "h": 7,
            "w": 8,
            "x": 16,
            "y": 12
          },
          "id": 102,
          "legend": {
            "avg": false,
            "current": false,
            "max": false,
            "min": false,
            "show": true,
            "total": false,
            "values": false
          },
          "lines": true,
          "linewidth": 1,
          "links": [],
          "nullPointMode": "null",
          "percentage": false,
          "pointradius": 5,
          "points": false,
          "renderer": "flot",
          "seriesOverrides": [
            {
              "alias": "/.*_count_.*/",
              "yaxis": 2
            }
          ],
          "spaceLength": 10,
          "stack": false,
          "steppedLine": false,
          "targets": [
            {
              "expr": "rate([[app]]_fuseclient_StreamWrite_count{app=\"$app\",cluster=\"$cluster\"}[1m])",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "StreamWrite_{{instance}}",
              "refId": "A"
            },
            {
              "expr": "rate([[app]]_fuseclient_StreamRead_count{app=\"$app\",cluster=\"$cluster\"}[1m])",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "StreamRead",
              "refId": "B"
            },
            {
              "expr": "rate([[app]]_fuseclient_StreamSyncWrite_count{instance=\"$instance\",cluster=\"$cluster\"}[1m])",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "SyncWrite",
              "refId": "C"
            },
            {
              "expr": "rate([[app]]_fuseclient_StreamSyncOverwrite_count{instance=\"$instance\",cluster=\"$cluster\"}[1m])",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "SyncOverwrite",
              "refId": "D"
            },
            {
              "expr": "rate([[app]]_fuseclient_StreamOverwrite_count{instance=\"$instance\",cluster=\"$cluster\"}[1m])",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "Overwrite",
              "refId": "E"
            },
            {
              "expr": "rate([[app]]_fuseclient_OpMetaExtentsList_count{app=\"$app\",cluster=\"$cluster\", instance=\"$instance\"}[1m])",
              "format": "time_series",
              "intervalFactor": 1,
              "legendFormat": "OpMetaExtentsList_count_{{instance}}",
              "refId": "F"
            },
            {
              "expr": "[[app]]_fuseclient_OpMetaExtentsList{app=\"$app\",cluster=\"$cluster\", instance=\"$instance\"}",
              "format": "time_series",
              "intervalFactor": 1,
              "legendFormat": "OpMetaExtentsList_{{instance}}",
              "refId": "G"
            }
          ],
          "thresholds": [],
          "timeFrom": null,
          "timeShift": null,
          "title": "MetaExtentsList",
          "tooltip": {
            "shared": true,
            "sort": 0,
            "value_type": "individual"
          },
          "type": "graph",
          "xaxis": {
            "buckets": null,
            "mode": "time",
            "name": null,
            "show": true,
            "values": []
          },
          "yaxes": [
            {
              "format": "locale",
              "label": null,
              "logBase": 1,
              "max": null,
              "min": null,
              "show": true
            },
            {
              "format": "ops",
              "label": null,
              "logBase": 1,
              "max": null,
              "min": null,
              "show": true
            }
          ],
          "yaxis": {
            "align": false,
            "alignLevel": null
          }
        },
        {
          "aliasColors": {},
          "bars": false,
          "dashLength": 10,
          "dashes": false,
          "datasource": null,
          "fill": 1,
          "gridPos": {
            "h": 7,
            "w": 8,
            "x": 0,
            "y": 19
          },
          "id": 69,
          "legend": {
            "avg": false,
            "current": false,
            "max": false,
            "min": false,
            "show": true,
            "total": false,
            "values": false
          },
          "lines": true,
          "linewidth": 1,
          "links": [],
          "nullPointMode": "null",
          "percentage": false,
          "pointradius": 5,
          "points": false,
          "renderer": "flot",
          "seriesOverrides": [
            {
              "alias": "/.*_count_.*/",
              "yaxis": 2
            }
          ],
          "spaceLength": 10,
          "stack": false,
          "steppedLine": false,
          "targets": [
            {
              "expr": "[[app]]_fuseclient_OpMetaExtentsAdd{app=~\"$app\", cluster=\"$cluster\", instance=\"$instance\"}",
              "format": "time_series",
              "intervalFactor": 1,
              "legendFormat": "OpMetaExtentsAdd_{{instance}}",
              "refId": "H"
            },
            {
              "expr": "rate([[app]]_fuseclient_OpMetaExtentsAdd_count{app=~\"$app\", cluster=\"$cluster\", instance=\"$instance\"}[1m])",
              "format": "time_series",
              "intervalFactor": 1,
              "legendFormat": "OpMetaExtentsAdd_count_{{instance}}",
              "refId": "A"
            },
            {
              "expr": "[[app]]_fuseclient_OpMetaExtentsList{instance=~\"$instance\"}",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "GetExtents",
              "refId": "I"
            }
          ],
          "thresholds": [],
          "timeFrom": null,
          "timeShift": null,
          "title": "MetaExtentsAdd",
          "tooltip": {
            "shared": true,
            "sort": 0,
            "value_type": "individual"
          },
          "type": "graph",
          "xaxis": {
            "buckets": null,
            "mode": "time",
            "name": null,
            "show": true,
            "values": []
          },
          "yaxes": [
            {
              "format": "ns",
              "label": null,
              "logBase": 1,
              "max": null,
              "min": null,
              "show": true
            },
            {
              "format": "ops",
              "label": null,
              "logBase": 1,
              "max": null,
              "min": null,
              "show": true
            }
          ],
          "yaxis": {
            "align": false,
            "alignLevel": null
          }
        },
        {
          "aliasColors": {},
          "bars": false,
          "dashLength": 10,
          "dashes": false,
          "datasource": null,
          "fill": 1,
          "gridPos": {
            "h": 7,
            "w": 8,
            "x": 8,
            "y": 19
          },
          "id": 148,
          "legend": {
            "avg": false,
            "current": false,
            "max": false,
            "min": false,
            "show": true,
            "total": false,
            "values": false
          },
          "lines": true,
          "linewidth": 1,
          "links": [],
          "nullPointMode": "null",
          "percentage": false,
          "pointradius": 5,
          "points": false,
          "renderer": "flot",
          "seriesOverrides": [
            {
              "alias": "/.*_ops_.*/",
              "yaxis": 2
            }
          ],
          "spaceLength": 10,
          "stack": false,
          "steppedLine": false,
          "targets": [
            {
              "expr": "[[app]]_fuseclient_OpMetaOpen{app=\"$app\", cluster=\"$cluster\"}",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "MetaOpen_{{instance}}",
              "refId": "B"
            },
            {
              "expr": "[[app]]_fuseclient_OpMetaReadDir{app=\"$app\", cluster=\"$cluster\"}",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "MetaReaddir_{{instance}}",
              "refId": "K"
            },
            {
              "expr": "[[app]]_fuseclient_OpMetaSetattr{app=\"$app\", cluster=\"$cluster\"}",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "MetaSetattr_{{instance}}",
              "refId": "L"
            },
            {
              "expr": "[[app]]_fuseclient_OpMetaLookup{app=\"$app\", cluster=\"$cluster\", instance=\"$instance\"}",
              "format": "time_series",
              "hide": false,
              "intervalFactor": 1,
              "legendFormat": "MetaLookup_{{instance}}",
              "refId": "A"
            },
            {
              "expr": "[[app]]_fuseclient_OpMetaTruncate{app=\"$app\", cluster=\"$cluster\"}",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "MetaTruncate_{{instance}}",
              "refId": "C"
            },
            {
              "expr": "rate([[app]]_fuseclient_OpMetaOpen_count{app=\"$app\", cluster=\"$cluster\"}[1m])",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "MetaOpen_ops_{{instance}}",
              "refId": "D"
            },
            {
              "expr": "rate([[app]]_fuseclient_OpMetaReadDir{app=\"$app\", cluster=\"$cluster\"}[1m])",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "MetaReaddir_ops_{{instance}}",
              "refId": "E"
            },
            {
              "expr": "rate([[app]]_fuseclient_OpMetaSetattr{app=\"$app\", cluster=\"$cluster\"}[1m])",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "MetaSetattr_ops_{{instance}}",
              "refId": "F"
            },
            {
              "expr": "rate([[app]]_fuseclient_OpMetaLookup_count{app=\"$app\", cluster=\"$cluster\", instance=\"$instance\"}[1m])",
              "format": "time_series",
              "hide": false,
              "intervalFactor": 1,
              "legendFormat": "MetaLookup_ops_{{instance}}",
              "refId": "G"
            },
            {
              "expr": "rate([[app]]_fuseclient_OpMetaTruncate{app=\"$app\", cluster=\"$cluster\"}[1m])",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "MetaTruncate_ops_{{instance}}",
              "refId": "H"
            }
          ],
          "thresholds": [],
          "timeFrom": null,
          "timeShift": null,
          "title": "MetaLookup",
          "tooltip": {
            "shared": true,
            "sort": 0,
            "value_type": "individual"
          },
          "type": "graph",
          "xaxis": {
            "buckets": null,
            "mode": "time",
            "name": null,
            "show": true,
            "values": []
          },
          "yaxes": [
            {
              "format": "ns",
              "label": null,
              "logBase": 1,
              "max": null,
              "min": null,
              "show": true
            },
            {
              "format": "ops",
              "label": null,
              "logBase": 1,
              "max": null,
              "min": null,
              "show": true
            }
          ],
          "yaxis": {
            "align": false,
            "alignLevel": null
          }
        },
        {
          "aliasColors": {},
          "bars": false,
          "dashLength": 10,
          "dashes": false,
          "datasource": null,
          "fill": 1,
          "gridPos": {
            "h": 7,
            "w": 8,
            "x": 16,
            "y": 19
          },
          "id": 67,
          "legend": {
            "avg": false,
            "current": false,
            "max": false,
            "min": false,
            "show": true,
            "total": false,
            "values": false
          },
          "lines": true,
          "linewidth": 1,
          "links": [],
          "nullPointMode": "null",
          "percentage": false,
          "pointradius": 5,
          "points": false,
          "renderer": "flot",
          "seriesOverrides": [
            {
              "alias": "/.*_count_.*/",
              "yaxis": 2
            }
          ],
          "spaceLength": 10,
          "stack": false,
          "steppedLine": false,
          "targets": [
            {
              "expr": "[[app]]_fuseclient_OpMetaBatchInodeGet{app=~\"$app\", cluster=\"$cluster\", instance=\"$instance\"}",
              "format": "time_series",
              "hide": false,
              "intervalFactor": 1,
              "legendFormat": "OpMetaBatchInodeGet_{{instance}}",
              "refId": "A"
            },
            {
              "expr": "[[app]]_fuseclient_OpMetaCreateInode{app=~\"$app\", cluster=\"$cluster\"}",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "CreateInode_{{instance}}",
              "refId": "D"
            },
            {
              "expr": "[[app]]_fuseclient_OpMetaDeleteInode{app=~\"$app\", cluster=\"$cluster\"}",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "DeleteInode_{{instance}}",
              "refId": "F"
            },
            {
              "expr": "[[app]]_fuseclient_OpMetaEvictInode{app=~\"$app\", cluster=\"$cluster\"}",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "EvictInode_{{instance}}",
              "refId": "G"
            },
            {
              "expr": "[[app]]_fuseclient_OpMetaInodeGet{app=~\"$app\", cluster=\"$cluster\"}",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "InodeGet_{{instance}}",
              "refId": "J"
            },
            {
              "expr": "[[app]]_fuseclient_OpMetaTruncate{app=~\"$app\", cluster=\"$cluster\"}",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "Truncate_{{instance}}",
              "refId": "B"
            },
            {
              "expr": "rate([[app]]_fuseclient_OpMetaBatchInodeGet_count{app=~\"$app\", cluster=\"$cluster\", instance=\"$instance\"}[1m])",
              "format": "time_series",
              "hide": false,
              "intervalFactor": 1,
              "legendFormat": "OpMetaBatchInodeGet_count_{{instance}}",
              "refId": "C"
            },
            {
              "expr": "rate([[app]]_fuseclient_OpMetaCreateInode{app=~\"$app\", cluster=\"$cluster\"}[1m])",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "CreateInode_count_{{instance}}",
              "refId": "E"
            },
            {
              "expr": "rate([[app]]_fuseclient_OpMetaDeleteInode{app=~\"$app\", cluster=\"$cluster\"}[1m])",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "DeleteInode_count_{{instance}}",
              "refId": "H"
            },
            {
              "expr": "rate([[app]]_fuseclient_OpMetaEvictInode{app=~\"$app\", cluster=\"$cluster\"}[1m])",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "EvictInode_count_{{instance}}",
              "refId": "I"
            },
            {
              "expr": "rate([[app]]_fuseclient_OpMetaInodeGet{app=~\"$app\", cluster=\"$cluster\"}[1m])",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "InodeGet_count_{{instance}}",
              "refId": "K"
            },
            {
              "expr": "rate([[app]]_fuseclient_OpMetaTruncate{app=~\"$app\", cluster=\"$cluster\"}[1m])",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "Truncate_count_{{instance}}",
              "refId": "L"
            }
          ],
          "thresholds": [],
          "timeFrom": null,
          "timeShift": null,
          "title": "MetaBatchInodeGet",
          "tooltip": {
            "shared": true,
            "sort": 0,
            "value_type": "individual"
          },
          "type": "graph",
          "xaxis": {
            "buckets": null,
            "mode": "time",
            "name": null,
            "show": true,
            "values": []
          },
          "yaxes": [
            {
              "format": "ns",
              "label": null,
              "logBase": 1,
              "max": null,
              "min": null,
              "show": true
            },
            {
              "format": "ops",
              "label": null,
              "logBase": 1,
              "max": null,
              "min": null,
              "show": true
            }
          ],
          "yaxis": {
            "align": false,
            "alignLevel": null
          }
        },
        {
          "aliasColors": {},
          "bars": false,
          "dashLength": 10,
          "dashes": false,
          "datasource": "default",
          "fill": 1,
          "gridPos": {
            "h": 7,
            "w": 8,
            "x": 0,
            "y": 26
          },
          "id": 152,
          "legend": {
            "avg": false,
            "current": false,
            "max": false,
            "min": false,
            "show": true,
            "total": false,
            "values": false
          },
          "lines": true,
          "linewidth": 1,
          "links": [],
          "nullPointMode": "null",
          "percentage": false,
          "pointradius": 5,
          "points": false,
          "renderer": "flot",
          "seriesOverrides": [
            {
              "alias": "/.*_count_.*/",
              "yaxis": 2
            }
          ],
          "spaceLength": 10,
          "stack": false,
          "steppedLine": false,
          "targets": [
            {
              "expr": "[[app]]_fuseclient_filecreate{app=~\"$app\",cluster=\"$cluster\", instance=\"$instance\"}",
              "format": "time_series",
              "intervalFactor": 1,
              "legendFormat": "filecreate_{{instance}}",
              "refId": "A"
            },
            {
              "expr": "rate([[app]]_fuseclient_filecreate_count{app=~\"$app\",cluster=\"$cluster\", instance=\"$instance\"}[1m])",
              "format": "time_series",
              "intervalFactor": 1,
              "legendFormat": "filecreate_count_{{instance}}",
              "refId": "B"
            },
            {
              "expr": "[[app]]_fuseclient_filesync{app=~\"$app\",cluster=\"$cluster\"}",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "filesync_{{instance}}",
              "refId": "C"
            },
            {
              "expr": "rate([[app]]_fuseclient_filesync_count{app=~\"$app\",cluster=\"$cluster\"}[1m])",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "filesync_count_{{instance}}",
              "refId": "D"
            },
            {
              "expr": "[[app]]_fuseclient_filewrite{app=~\"$app\",cluster=\"$cluster\"}",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "filewrite_{{instance}}",
              "refId": "E"
            },
            {
              "expr": "rate([[app]]_fuseclient_filewrite_count{app=~\"$app\",cluster=\"$cluster\"}[1m])",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "filewrite_count_{{instance}}",
              "refId": "F"
            }
          ],
          "thresholds": [],
          "timeFrom": null,
          "timeShift": null,
          "title": "FileCreate",
          "tooltip": {
            "shared": true,
            "sort": 0,
            "value_type": "individual"
          },
          "type": "graph",
          "xaxis": {
            "buckets": null,
            "mode": "time",
            "name": null,
            "show": true,
            "values": []
          },
          "yaxes": [
            {
              "format": "ns",
              "label": null,
              "logBase": 1,
              "max": null,
              "min": null,
              "show": true
            },
            {
              "format": "ops",
              "label": null,
              "logBase": 1,
              "max": null,
              "min": null,
              "show": true
            }
          ],
          "yaxis": {
            "align": false,
            "alignLevel": null
          }
        },
        {
          "aliasColors": {},
          "bars": false,
          "dashLength": 10,
          "dashes": false,
          "datasource": null,
          "fill": 1,
          "gridPos": {
            "h": 7,
            "w": 8,
            "x": 8,
            "y": 26
          },
          "id": 158,
          "legend": {
            "avg": false,
            "current": false,
            "max": false,
            "min": false,
            "show": true,
            "total": false,
            "values": false
          },
          "lines": true,
          "linewidth": 1,
          "links": [],
          "nullPointMode": "null",
          "percentage": false,
          "pointradius": 5,
          "points": false,
          "renderer": "flot",
          "seriesOverrides": [
            {
              "alias": "/.*_count_.*/",
              "yaxis": 2
            }
          ],
          "spaceLength": 10,
          "stack": false,
          "steppedLine": false,
          "targets": [
            {
              "expr": "[[app]]_fuseclient_OpMetaInodeGet{app=~\"$app\", cluster=\"$cluster\", instance=\"$instance\"}",
              "format": "time_series",
              "hide": false,
              "intervalFactor": 1,
              "legendFormat": "InodeGet_{{instance}}",
              "refId": "J"
            },
            {
              "expr": "rate([[app]]_fuseclient_OpMetaInodeGet_count{app=~\"$app\", cluster=\"$cluster\", instance=\"$instance\"}[1m])",
              "format": "time_series",
              "hide": false,
              "intervalFactor": 1,
              "legendFormat": "InodeGet_count_{{instance}}",
              "refId": "K"
            }
          ],
          "thresholds": [],
          "timeFrom": null,
          "timeShift": null,
          "title": "InodeGet",
          "tooltip": {
            "shared": true,
            "sort": 0,
            "value_type": "individual"
          },
          "type": "graph",
          "xaxis": {
            "buckets": null,
            "mode": "time",
            "name": null,
            "show": true,
            "values": []
          },
          "yaxes": [
            {
              "format": "ns",
              "label": null,
              "logBase": 1,
              "max": null,
              "min": null,
              "show": true
            },
            {
              "format": "ops",
              "label": null,
              "logBase": 1,
              "max": null,
              "min": null,
              "show": true
            }
          ],
          "yaxis": {
            "align": false,
            "alignLevel": null
          }
        },
        {
          "aliasColors": {},
          "bars": false,
          "dashLength": 10,
          "dashes": false,
          "datasource": null,
          "fill": 1,
          "gridPos": {
            "h": 7,
            "w": 8,
            "x": 16,
            "y": 26
          },
          "id": 155,
          "legend": {
            "avg": false,
            "current": false,
            "max": false,
            "min": false,
            "show": true,
            "total": false,
            "values": false
          },
          "lines": true,
          "linewidth": 1,
          "links": [],
          "nullPointMode": "null",
          "percentage": false,
          "pointradius": 5,
          "points": false,
          "renderer": "flot",
          "seriesOverrides": [
            {
              "alias": "/.*_count_.*/",
              "yaxis": 2
            }
          ],
          "spaceLength": 10,
          "stack": false,
          "steppedLine": false,
          "targets": [
            {
              "expr": "[[app]]_fuseclient_OpMetaBatchInodeGet{app=~\"$app\", cluster=\"$cluster\"}",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "OpMetaBatchInodeGet_{{instance}}",
              "refId": "A"
            },
            {
              "expr": "[[app]]_fuseclient_OpMetaCreateInode{app=~\"$app\", cluster=\"$cluster\"}",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "CreateInode_{{instance}}",
              "refId": "D"
            },
            {
              "expr": "[[app]]_fuseclient_OpMetaDeleteInode{app=~\"$app\", cluster=\"$cluster\"}",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "DeleteInode_{{instance}}",
              "refId": "F"
            },
            {
              "expr": "[[app]]_fuseclient_OpMetaEvictInode{app=~\"$app\", cluster=\"$cluster\", instance=\"$instance\"}",
              "format": "time_series",
              "hide": false,
              "intervalFactor": 1,
              "legendFormat": "EvictInode_{{instance}}",
              "refId": "G"
            },
            {
              "expr": "[[app]]_fuseclient_OpMetaInodeGet{app=~\"$app\", cluster=\"$cluster\"}",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "InodeGet_{{instance}}",
              "refId": "J"
            },
            {
              "expr": "[[app]]_fuseclient_OpMetaTruncate{app=~\"$app\", cluster=\"$cluster\"}",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "Truncate_{{instance}}",
              "refId": "B"
            },
            {
              "expr": "rate([[app]]_fuseclient_OpMetaBatchInodeGet_count{app=~\"$app\", cluster=\"$cluster\"}[1m])",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "OpMetaBatchInodeGet_count_{{instance}}",
              "refId": "C"
            },
            {
              "expr": "rate([[app]]_fuseclient_OpMetaCreateInode{app=~\"$app\", cluster=\"$cluster\"}[1m])",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "CreateInode_count_{{instance}}",
              "refId": "E"
            },
            {
              "expr": "rate([[app]]_fuseclient_OpMetaDeleteInode{app=~\"$app\", cluster=\"$cluster\"}[1m])",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "DeleteInode_count_{{instance}}",
              "refId": "H"
            },
            {
              "expr": "rate([[app]]_fuseclient_OpMetaEvictInode_count{app=~\"$app\", cluster=\"$cluster\", instance=\"$instance\"}[1m])",
              "format": "time_series",
              "hide": false,
              "intervalFactor": 1,
              "legendFormat": "EvictInode_count_{{instance}}",
              "refId": "I"
            },
            {
              "expr": "rate([[app]]_fuseclient_OpMetaInodeGet{app=~\"$app\", cluster=\"$cluster\"}[1m])",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "InodeGet_count_{{instance}}",
              "refId": "K"
            },
            {
              "expr": "rate([[app]]_fuseclient_OpMetaTruncate{app=~\"$app\", cluster=\"$cluster\"}[1m])",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "Truncate_count_{{instance}}",
              "refId": "L"
            }
          ],
          "thresholds": [],
          "timeFrom": null,
          "timeShift": null,
          "title": "EvictInode",
          "tooltip": {
            "shared": true,
            "sort": 0,
            "value_type": "individual"
          },
          "type": "graph",
          "xaxis": {
            "buckets": null,
            "mode": "time",
            "name": null,
            "show": true,
            "values": []
          },
          "yaxes": [
            {
              "format": "ns",
              "label": null,
              "logBase": 1,
              "max": null,
              "min": null,
              "show": true
            },
            {
              "format": "ops",
              "label": null,
              "logBase": 1,
              "max": null,
              "min": null,
              "show": true
            }
          ],
          "yaxis": {
            "align": false,
            "alignLevel": null
          }
        },
        {
          "aliasColors": {},
          "bars": false,
          "dashLength": 10,
          "dashes": false,
          "datasource": "default",
          "fill": 1,
          "gridPos": {
            "h": 7,
            "w": 8,
            "x": 0,
            "y": 33
          },
          "id": 153,
          "legend": {
            "avg": false,
            "current": false,
            "max": false,
            "min": false,
            "show": true,
            "total": false,
            "values": false
          },
          "lines": true,
          "linewidth": 1,
          "links": [],
          "nullPointMode": "null",
          "percentage": false,
          "pointradius": 5,
          "points": false,
          "renderer": "flot",
          "seriesOverrides": [
            {
              "alias": "/.*_count_.*/",
              "yaxis": 2
            }
          ],
          "spaceLength": 10,
          "stack": false,
          "steppedLine": false,
          "targets": [
            {
              "expr": "[[app]]_fuseclient_filecreate{app=~\"$app\",cluster=\"$cluster\"}",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "filecreate_{{instance}}",
              "refId": "A"
            },
            {
              "expr": "rate([[app]]_fuseclient_filecreate_count{app=~\"$app\",cluster=\"$cluster\"}[1m])",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "filecreate_count_{{instance}}",
              "refId": "B"
            },
            {
              "expr": "[[app]]_fuseclient_filesync{app=~\"$app\",cluster=\"$cluster\", instance=\"$instance\"}",
              "format": "time_series",
              "hide": false,
              "intervalFactor": 1,
              "legendFormat": "filesync_{{instance}}",
              "refId": "C"
            },
            {
              "expr": "rate([[app]]_fuseclient_filesync_count{app=~\"$app\",cluster=\"$cluster\", instance=\"$instance\"}[1m])",
              "format": "time_series",
              "hide": false,
              "intervalFactor": 1,
              "legendFormat": "filesync_count_{{instance}}",
              "refId": "D"
            },
            {
              "expr": "[[app]]_fuseclient_filewrite{app=~\"$app\",cluster=\"$cluster\"}",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "filewrite_{{instance}}",
              "refId": "E"
            },
            {
              "expr": "rate([[app]]_fuseclient_filewrite_count{app=~\"$app\",cluster=\"$cluster\"}[1m])",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "filewrite_count_{{instance}}",
              "refId": "F"
            }
          ],
          "thresholds": [],
          "timeFrom": null,
          "timeShift": null,
          "title": "FileSync",
          "tooltip": {
            "shared": true,
            "sort": 0,
            "value_type": "individual"
          },
          "type": "graph",
          "xaxis": {
            "buckets": null,
            "mode": "time",
            "name": null,
            "show": true,
            "values": []
          },
          "yaxes": [
            {
              "format": "ns",
              "label": null,
              "logBase": 1,
              "max": null,
              "min": null,
              "show": true
            },
            {
              "format": "ops",
              "label": null,
              "logBase": 1,
              "max": null,
              "min": null,
              "show": true
            }
          ],
          "yaxis": {
            "align": false,
            "alignLevel": null
          }
        },
        {
          "aliasColors": {},
          "bars": false,
          "dashLength": 10,
          "dashes": false,
          "datasource": "default",
          "fill": 1,
          "gridPos": {
            "h": 7,
            "w": 8,
            "x": 8,
            "y": 33
          },
          "id": 100,
          "legend": {
            "avg": false,
            "current": false,
            "max": false,
            "min": false,
            "show": true,
            "total": false,
            "values": false
          },
          "lines": true,
          "linewidth": 1,
          "links": [],
          "nullPointMode": "null",
          "percentage": false,
          "pointradius": 5,
          "points": false,
          "renderer": "flot",
          "seriesOverrides": [
            {
              "alias": "/.*_count_.*/",
              "yaxis": 2
            }
          ],
          "spaceLength": 10,
          "stack": false,
          "steppedLine": false,
          "targets": [
            {
              "expr": "[[app]]_fuseclient_fileread{app=~\"$app\",cluster=\"$cluster\", instance=\"$instance\"}",
              "format": "time_series",
              "intervalFactor": 1,
              "legendFormat": "fileread_{{instance}}",
              "refId": "A"
            },
            {
              "expr": "rate([[app]]_fuseclient_fileread_count{app=~\"$app\",cluster=\"$cluster\", instance=\"$instance\"}[1m])",
              "format": "time_series",
              "intervalFactor": 1,
              "legendFormat": "fileread_count_{{instance}}",
              "refId": "B"
            }
          ],
          "thresholds": [],
          "timeFrom": null,
          "timeShift": null,
          "title": "FileRead",
          "tooltip": {
            "shared": true,
            "sort": 0,
            "value_type": "individual"
          },
          "type": "graph",
          "xaxis": {
            "buckets": null,
            "mode": "time",
            "name": null,
            "show": true,
            "values": []
          },
          "yaxes": [
            {
              "format": "ns",
              "label": null,
              "logBase": 1,
              "max": null,
              "min": null,
              "show": true
            },
            {
              "format": "ops",
              "label": null,
              "logBase": 1,
              "max": null,
              "min": null,
              "show": true
            }
          ],
          "yaxis": {
            "align": false,
            "alignLevel": null
          }
        },
        {
          "aliasColors": {},
          "bars": false,
          "dashLength": 10,
          "dashes": false,
          "datasource": "default",
          "fill": 1,
          "gridPos": {
            "h": 7,
            "w": 8,
            "x": 16,
            "y": 33
          },
          "id": 154,
          "legend": {
            "avg": false,
            "current": false,
            "max": false,
            "min": false,
            "show": true,
            "total": false,
            "values": false
          },
          "lines": true,
          "linewidth": 1,
          "links": [],
          "nullPointMode": "null",
          "percentage": false,
          "pointradius": 5,
          "points": false,
          "renderer": "flot",
          "seriesOverrides": [
            {
              "alias": "/.*_count_.*/",
              "yaxis": 2
            }
          ],
          "spaceLength": 10,
          "stack": false,
          "steppedLine": false,
          "targets": [
            {
              "expr": "[[app]]_fuseclient_filecreate{app=~\"$app\",cluster=\"$cluster\"}",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "filecreate_{{instance}}",
              "refId": "A"
            },
            {
              "expr": "rate([[app]]_fuseclient_filecreate_count{app=~\"$app\",cluster=\"$cluster\"}[1m])",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "filecreate_count_{{instance}}",
              "refId": "B"
            },
            {
              "expr": "[[app]]_fuseclient_filesync{app=~\"$app\",cluster=\"$cluster\"}",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "filesync_{{instance}}",
              "refId": "C"
            },
            {
              "expr": "rate([[app]]_fuseclient_filesync_count{app=~\"$app\",cluster=\"$cluster\"}[1m])",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "filesync_count_{{instance}}",
              "refId": "D"
            },
            {
              "expr": "[[app]]_fuseclient_filewrite{app=~\"$app\",cluster=\"$cluster\", instance=\"$instance\"}",
              "format": "time_series",
              "hide": false,
              "intervalFactor": 1,
              "legendFormat": "filewrite_{{instance}}",
              "refId": "E"
            },
            {
              "expr": "rate([[app]]_fuseclient_filewrite_count{app=~\"$app\",cluster=\"$cluster\", instance=\"$instance\"}[1m])",
              "format": "time_series",
              "hide": false,
              "intervalFactor": 1,
              "legendFormat": "filewrite_count_{{instance}}",
              "refId": "F"
            }
          ],
          "thresholds": [],
          "timeFrom": null,
          "timeShift": null,
          "title": "FileWrite",
          "tooltip": {
            "shared": true,
            "sort": 0,
            "value_type": "individual"
          },
          "type": "graph",
          "xaxis": {
            "buckets": null,
            "mode": "time",
            "name": null,
            "show": true,
            "values": []
          },
          "yaxes": [
            {
              "format": "ns",
              "label": null,
              "logBase": 1,
              "max": null,
              "min": null,
              "show": true
            },
            {
              "format": "ops",
              "label": null,
              "logBase": 1,
              "max": null,
              "min": null,
              "show": true
            }
          ],
          "yaxis": {
            "align": false,
            "alignLevel": null
          }
        },
        {
          "aliasColors": {},
          "bars": false,
          "dashLength": 10,
          "dashes": false,
          "datasource": null,
          "fill": 1,
          "gridPos": {
            "h": 7,
            "w": 8,
            "x": 0,
            "y": 40
          },
          "id": 147,
          "legend": {
            "avg": false,
            "current": false,
            "max": false,
            "min": false,
            "show": true,
            "total": false,
            "values": false
          },
          "lines": true,
          "linewidth": 1,
          "links": [],
          "nullPointMode": "null",
          "percentage": false,
          "pointradius": 5,
          "points": false,
          "renderer": "flot",
          "seriesOverrides": [
            {
              "alias": "/.*_ops_.*/",
              "yaxis": 2
            }
          ],
          "spaceLength": 10,
          "stack": false,
          "steppedLine": false,
          "targets": [
            {
              "expr": "[[app]]_fuseclient_OpMetaOpen{app=\"$app\", cluster=\"$cluster\"}",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "MetaOpen_{{instance}}",
              "refId": "B"
            },
            {
              "expr": "[[app]]_fuseclient_OpMetaReadDir{app=\"$app\", cluster=\"$cluster\"}",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "MetaReaddir_{{instance}}",
              "refId": "K"
            },
            {
              "expr": "[[app]]_fuseclient_OpMetaSetattr{app=\"$app\", cluster=\"$cluster\"}",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "MetaSetattr_{{instance}}",
              "refId": "L"
            },
            {
              "expr": "[[app]]_fuseclient_OpMetaLookup{app=\"$app\", cluster=\"$cluster\"}",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "MetaLookup_{{instance}}",
              "refId": "A"
            },
            {
              "expr": "[[app]]_fuseclient_OpMetaTruncate{app=\"$app\", cluster=\"$cluster\", instance=\"$instance\"}",
              "format": "time_series",
              "hide": false,
              "intervalFactor": 1,
              "legendFormat": "MetaTruncate_{{instance}}",
              "refId": "C"
            },
            {
              "expr": "rate([[app]]_fuseclient_OpMetaOpen_count{app=\"$app\", cluster=\"$cluster\"}[1m])",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "MetaOpen_ops_{{instance}}",
              "refId": "D"
            },
            {
              "expr": "rate([[app]]_fuseclient_OpMetaReadDir{app=\"$app\", cluster=\"$cluster\"}[1m])",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "MetaReaddir_ops_{{instance}}",
              "refId": "E"
            },
            {
              "expr": "rate([[app]]_fuseclient_OpMetaSetattr{app=\"$app\", cluster=\"$cluster\"}[1m])",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "MetaSetattr_ops_{{instance}}",
              "refId": "F"
            },
            {
              "expr": "rate([[app]]_fuseclient_OpMetaLookup{app=\"$app\", cluster=\"$cluster\"}[1m])",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "MetaLookup_ops_{{instance}}",
              "refId": "G"
            },
            {
              "expr": "rate([[app]]_fuseclient_OpMetaTruncate_count{app=\"$app\", cluster=\"$cluster\", instance=\"$instance\"}[1m])",
              "format": "time_series",
              "hide": false,
              "intervalFactor": 1,
              "legendFormat": "MetaTruncate_ops_{{instance}}",
              "refId": "H"
            }
          ],
          "thresholds": [],
          "timeFrom": null,
          "timeShift": null,
          "title": "MetaTruncate",
          "tooltip": {
            "shared": true,
            "sort": 0,
            "value_type": "individual"
          },
          "type": "graph",
          "xaxis": {
            "buckets": null,
            "mode": "time",
            "name": null,
            "show": true,
            "values": []
          },
          "yaxes": [
            {
              "format": "ns",
              "label": null,
              "logBase": 1,
              "max": null,
              "min": null,
              "show": true
            },
            {
              "format": "ops",
              "label": null,
              "logBase": 1,
              "max": null,
              "min": null,
              "show": true
            }
          ],
          "yaxis": {
            "align": false,
            "alignLevel": null
          }
        },
        {
          "aliasColors": {},
          "bars": false,
          "dashLength": 10,
          "dashes": false,
          "datasource": null,
          "fill": 1,
          "gridPos": {
            "h": 7,
            "w": 8,
            "x": 8,
            "y": 40
          },
          "id": 149,
          "legend": {
            "avg": false,
            "current": false,
            "max": false,
            "min": false,
            "show": true,
            "total": false,
            "values": false
          },
          "lines": true,
          "linewidth": 1,
          "links": [],
          "nullPointMode": "null",
          "percentage": false,
          "pointradius": 5,
          "points": false,
          "renderer": "flot",
          "seriesOverrides": [
            {
              "alias": "/.*_ops_.*/",
              "yaxis": 2
            }
          ],
          "spaceLength": 10,
          "stack": false,
          "steppedLine": false,
          "targets": [
            {
              "expr": "[[app]]_fuseclient_OpMetaOpen{app=\"$app\", cluster=\"$cluster\"}",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "MetaOpen_{{instance}}",
              "refId": "B"
            },
            {
              "expr": "[[app]]_fuseclient_OpMetaReadDir{app=\"$app\", cluster=\"$cluster\"}",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "MetaReaddir_{{instance}}",
              "refId": "K"
            },
            {
              "expr": "[[app]]_fuseclient_OpMetaSetattr{app=\"$app\", cluster=\"$cluster\", instance=\"$instance\"}",
              "format": "time_series",
              "hide": false,
              "intervalFactor": 1,
              "legendFormat": "MetaSetattr_{{instance}}",
              "refId": "L"
            },
            {
              "expr": "[[app]]_fuseclient_OpMetaLookup{app=\"$app\", cluster=\"$cluster\"}",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "MetaLookup_{{instance}}",
              "refId": "A"
            },
            {
              "expr": "[[app]]_fuseclient_OpMetaTruncate{app=\"$app\", cluster=\"$cluster\"}",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "MetaTruncate_{{instance}}",
              "refId": "C"
            },
            {
              "expr": "rate([[app]]_fuseclient_OpMetaOpen_count{app=\"$app\", cluster=\"$cluster\"}[1m])",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "MetaOpen_ops_{{instance}}",
              "refId": "D"
            },
            {
              "expr": "rate([[app]]_fuseclient_OpMetaReadDir{app=\"$app\", cluster=\"$cluster\"}[1m])",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "MetaReaddir_ops_{{instance}}",
              "refId": "E"
            },
            {
              "expr": "rate([[app]]_fuseclient_OpMetaSetattr_cout{app=\"$app\", cluster=\"$cluster\", instance=\"$instance\"}[1m])",
              "format": "time_series",
              "hide": false,
              "intervalFactor": 1,
              "legendFormat": "MetaSetattr_ops_{{instance}}",
              "refId": "F"
            },
            {
              "expr": "rate([[app]]_fuseclient_OpMetaLookup_count{app=\"$app\", cluster=\"$cluster\"}[1m])",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "MetaLookup_ops_{{instance}}",
              "refId": "G"
            },
            {
              "expr": "rate([[app]]_fuseclient_OpMetaTruncate_count{app=\"$app\", cluster=\"$cluster\"}[1m])",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "MetaTruncate_ops_{{instance}}",
              "refId": "H"
            }
          ],
          "thresholds": [],
          "timeFrom": null,
          "timeShift": null,
          "title": "MetaSetattr",
          "tooltip": {
            "shared": true,
            "sort": 0,
            "value_type": "individual"
          },
          "type": "graph",
          "xaxis": {
            "buckets": null,
            "mode": "time",
            "name": null,
            "show": true,
            "values": []
          },
          "yaxes": [
            {
              "format": "ns",
              "label": null,
              "logBase": 1,
              "max": null,
              "min": null,
              "show": true
            },
            {
              "format": "ops",
              "label": null,
              "logBase": 1,
              "max": null,
              "min": null,
              "show": true
            }
          ],
          "yaxis": {
            "align": false,
            "alignLevel": null
          }
        },
        {
          "aliasColors": {},
          "bars": false,
          "dashLength": 10,
          "dashes": false,
          "datasource": null,
          "fill": 1,
          "gridPos": {
            "h": 7,
            "w": 8,
            "x": 16,
            "y": 40
          },
          "id": 150,
          "legend": {
            "avg": false,
            "current": false,
            "max": false,
            "min": false,
            "show": true,
            "total": false,
            "values": false
          },
          "lines": true,
          "linewidth": 1,
          "links": [],
          "nullPointMode": "null",
          "percentage": false,
          "pointradius": 5,
          "points": false,
          "renderer": "flot",
          "seriesOverrides": [
            {
              "alias": "/.*_ops_.*/",
              "yaxis": 2
            }
          ],
          "spaceLength": 10,
          "stack": false,
          "steppedLine": false,
          "targets": [
            {
              "expr": "[[app]]_fuseclient_OpMetaOpen{app=\"$app\", cluster=\"$cluster\", instance=\"$instance\"}",
              "format": "time_series",
              "hide": false,
              "intervalFactor": 1,
              "legendFormat": "MetaOpen_{{instance}}",
              "refId": "B"
            },
            {
              "expr": "[[app]]_fuseclient_OpMetaReadDir{app=\"$app\", cluster=\"$cluster\"}",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "MetaReaddir_{{instance}}",
              "refId": "K"
            },
            {
              "expr": "[[app]]_fuseclient_OpMetaSetattr{app=\"$app\", cluster=\"$cluster\"}",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "MetaSetattr_{{instance}}",
              "refId": "L"
            },
            {
              "expr": "[[app]]_fuseclient_OpMetaLookup{app=\"$app\", cluster=\"$cluster\"}",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "MetaLookup_{{instance}}",
              "refId": "A"
            },
            {
              "expr": "[[app]]_fuseclient_OpMetaTruncate{app=\"$app\", cluster=\"$cluster\"}",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "MetaTruncate_{{instance}}",
              "refId": "C"
            },
            {
              "expr": "rate([[app]]_fuseclient_OpMetaOpen_count{app=\"$app\", cluster=\"$cluster\", instance=\"$instance\"}[1m])",
              "format": "time_series",
              "hide": false,
              "intervalFactor": 1,
              "legendFormat": "MetaOpen_ops_{{instance}}",
              "refId": "D"
            },
            {
              "expr": "rate([[app]]_fuseclient_OpMetaReadDir{app=\"$app\", cluster=\"$cluster\"}[1m])",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "MetaReaddir_ops_{{instance}}",
              "refId": "E"
            },
            {
              "expr": "rate([[app]]_fuseclient_OpMetaSetattr{app=\"$app\", cluster=\"$cluster\"}[1m])",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "MetaSetattr_ops_{{instance}}",
              "refId": "F"
            },
            {
              "expr": "rate([[app]]_fuseclient_OpMetaLookup{app=\"$app\", cluster=\"$cluster\"}[1m])",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "MetaLookup_ops_{{instance}}",
              "refId": "G"
            },
            {
              "expr": "rate([[app]]_fuseclient_OpMetaTruncate{app=\"$app\", cluster=\"$cluster\"}[1m])",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "MetaTruncate_ops_{{instance}}",
              "refId": "H"
            }
          ],
          "thresholds": [],
          "timeFrom": null,
          "timeShift": null,
          "title": "MetaOpen",
          "tooltip": {
            "shared": true,
            "sort": 0,
            "value_type": "individual"
          },
          "type": "graph",
          "xaxis": {
            "buckets": null,
            "mode": "time",
            "name": null,
            "show": true,
            "values": []
          },
          "yaxes": [
            {
              "format": "ns",
              "label": null,
              "logBase": 1,
              "max": null,
              "min": null,
              "show": true
            },
            {
              "format": "ops",
              "label": null,
              "logBase": 1,
              "max": null,
              "min": null,
              "show": true
            }
          ],
          "yaxis": {
            "align": false,
            "alignLevel": null
          }
        },
        {
          "aliasColors": {},
          "bars": false,
          "dashLength": 10,
          "dashes": false,
          "datasource": null,
          "fill": 1,
          "gridPos": {
            "h": 7,
            "w": 8,
            "x": 0,
            "y": 47
          },
          "id": 106,
          "legend": {
            "avg": false,
            "current": false,
            "max": false,
            "min": false,
            "show": true,
            "total": false,
            "values": false
          },
          "lines": true,
          "linewidth": 1,
          "links": [],
          "nullPointMode": "null",
          "percentage": false,
          "pointradius": 5,
          "points": false,
          "renderer": "flot",
          "seriesOverrides": [],
          "spaceLength": 10,
          "stack": false,
          "steppedLine": false,
          "targets": [
            {
              "expr": "[[app]]_fuseclient_FlushInRead{app=\"$app\", cluster=\"$cluster\", instance=\"$instance\"}",
              "format": "time_series",
              "intervalFactor": 1,
              "legendFormat": "FlushInRead_{{instance}}",
              "refId": "A"
            },
            {
              "expr": "[[app]]_fuseclient_StreamPrepareReadRequests{app=~\"$app\", cluster=\"$cluster\", instance=\"$instance\"}",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "PrepareReadRequest_{{instance}}",
              "refId": "B"
            }
          ],
          "thresholds": [],
          "timeFrom": null,
          "timeShift": null,
          "title": "FlushInRead",
          "tooltip": {
            "shared": true,
            "sort": 0,
            "value_type": "individual"
          },
          "type": "graph",
          "xaxis": {
            "buckets": null,
            "mode": "time",
            "name": null,
            "show": true,
            "values": []
          },
          "yaxes": [
            {
              "format": "ns",
              "label": null,
              "logBase": 1,
              "max": null,
              "min": null,
              "show": true
            },
            {
              "format": "short",
              "label": null,
              "logBase": 1,
              "max": null,
              "min": null,
              "show": true
            }
          ],
          "yaxis": {
            "align": false,
            "alignLevel": null
          }
        },
        {
          "aliasColors": {},
          "bars": false,
          "dashLength": 10,
          "dashes": false,
          "datasource": null,
          "fill": 1,
          "gridPos": {
            "h": 7,
            "w": 8,
            "x": 8,
            "y": 47
          },
          "id": 161,
          "legend": {
            "avg": false,
            "current": false,
            "max": false,
            "min": false,
            "show": true,
            "total": false,
            "values": false
          },
          "lines": true,
          "linewidth": 1,
          "links": [],
          "nullPointMode": "null",
          "percentage": false,
          "pointradius": 5,
          "points": false,
          "renderer": "flot",
          "seriesOverrides": [
            {
              "alias": "/.*_count_.*/",
              "yaxis": 2
            }
          ],
          "spaceLength": 10,
          "stack": false,
          "steppedLine": false,
          "targets": [
            {
              "expr": "[[app]]_fuseclient_OpMetaCreateDentry{app=~\"$app\", cluster=\"$cluster\"}",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "OpMetaCreateDentry_{{instance}}",
              "refId": "C"
            },
            {
              "expr": "[[app]]_fuseclient_OpMetaDeleteDentry{app=~\"$app\", cluster=\"$cluster\"}",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "OpMetaDeleteDentry_{{instance}}",
              "refId": "E"
            },
            {
              "expr": "[[app]]_fuseclient_OpMetaUpdateDentry{app=~\"$app\", cluster=\"$cluster\", instance=\"$instance\"}",
              "format": "time_series",
              "hide": false,
              "intervalFactor": 1,
              "legendFormat": "OpMetaUpdateDentry_{{instance}}",
              "refId": "A"
            },
            {
              "expr": "[[app]]_fuseclient_OpMetaCreateDentry_count{app=~\"$app\", cluster=\"$cluster\"}",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "OpMetaCreateDentry_count_{{instance}}",
              "refId": "B"
            },
            {
              "expr": "[[app]]_fuseclient_OpMetaDeleteDentry_count{app=~\"$app\", cluster=\"$cluster\"}",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "OpMetaDeleteDentry_count_{{instance}}",
              "refId": "D"
            },
            {
              "expr": "[[app]]_fuseclient_OpMetaUpdateDentry_count{app=~\"$app\", cluster=\"$cluster\", instance=\"$instance\"}",
              "format": "time_series",
              "hide": false,
              "intervalFactor": 1,
              "legendFormat": "OpMetaUpdateDentry_count_{{instance}}",
              "refId": "F"
            }
          ],
          "thresholds": [],
          "timeFrom": null,
          "timeShift": null,
          "title": "MetaUpdateDentry",
          "tooltip": {
            "shared": true,
            "sort": 0,
            "value_type": "individual"
          },
          "type": "graph",
          "xaxis": {
            "buckets": null,
            "mode": "time",
            "name": null,
            "show": true,
            "values": []
          },
          "yaxes": [
            {
              "format": "ns",
              "label": null,
              "logBase": 1,
              "max": null,
              "min": null,
              "show": true
            },
            {
              "format": "ops",
              "label": null,
              "logBase": 1,
              "max": null,
              "min": null,
              "show": true
            }
          ],
          "yaxis": {
            "align": false,
            "alignLevel": null
          }
        },
        {
          "aliasColors": {},
          "bars": false,
          "dashLength": 10,
          "dashes": false,
          "datasource": null,
          "fill": 1,
          "gridPos": {
            "h": 7,
            "w": 8,
            "x": 16,
            "y": 47
          },
          "id": 98,
          "legend": {
            "avg": false,
            "current": false,
            "max": false,
            "min": false,
            "show": true,
            "total": false,
            "values": false
          },
          "lines": true,
          "linewidth": 1,
          "links": [],
          "nullPointMode": "null",
          "percentage": false,
          "pointradius": 5,
          "points": false,
          "renderer": "flot",
          "seriesOverrides": [],
          "spaceLength": 10,
          "stack": false,
          "steppedLine": false,
          "targets": [
            {
              "expr": "[[app]]_fuseclient_StreamWrite{instance=~\"$instance\",cluster=\"$cluster\", instance=\"$instance\"}",
              "format": "time_series",
              "intervalFactor": 1,
              "legendFormat": "Write",
              "refId": "A"
            },
            {
              "expr": "[[app]]_fuseclient_StreamRead{instance=~\"$instance\",cluster=\"$cluster\", instance=\"$instance\"}",
              "format": "time_series",
              "intervalFactor": 1,
              "legendFormat": "Read",
              "refId": "C"
            },
            {
              "expr": "[[app]]_fuseclient_StreamOverwrite{instance=~\"$instance\",cluster=\"$cluster\", instance=\"$instance\"}",
              "format": "time_series",
              "intervalFactor": 1,
              "legendFormat": "Overwrite",
              "refId": "D"
            },
            {
              "expr": "[[app]]_fuseclient_StreamCreateExtent{instance=~\"$instance\",cluster=\"$cluster\", instance=\"$instance\"}",
              "format": "time_series",
              "intervalFactor": 1,
              "legendFormat": "CreateExtent",
              "refId": "B"
            },
            {
              "expr": "[[app]]_fuseclient_StreamSyncWrite{instance=~\"$instance\",cluster=\"$cluster\"}",
              "format": "time_series",
              "intervalFactor": 1,
              "legendFormat": "SyncWrite",
              "refId": "E"
            },
            {
              "expr": "[[app]]_fuseclient_StreamSyncOverwrite{instance=~\"$instance\",cluster=\"$cluster\"}",
              "format": "time_series",
              "intervalFactor": 1,
              "legendFormat": "SyncOverwrite",
              "refId": "F"
            }
          ],
          "thresholds": [],
          "timeFrom": null,
          "timeShift": null,
          "title": "StreamSDKLatency",
          "tooltip": {
            "shared": true,
            "sort": 0,
            "value_type": "individual"
          },
          "type": "graph",
          "xaxis": {
            "buckets": null,
            "mode": "time",
            "name": null,
            "show": true,
            "values": []
          },
          "yaxes": [
            {
              "format": "ns",
              "label": null,
              "logBase": 1,
              "max": null,
              "min": null,
              "show": true
            },
            {
              "format": "short",
              "label": null,
              "logBase": 1,
              "max": null,
              "min": null,
              "show": true
            }
          ],
          "yaxis": {
            "align": false,
            "alignLevel": null
          }
        },
        {
          "aliasColors": {},
          "bars": false,
          "dashLength": 10,
          "dashes": false,
          "datasource": null,
          "fill": 1,
          "gridPos": {
            "h": 7,
            "w": 8,
            "x": 0,
            "y": 54
          },
          "id": 159,
          "legend": {
            "avg": false,
            "current": false,
            "max": false,
            "min": false,
            "show": true,
            "total": false,
            "values": false
          },
          "lines": true,
          "linewidth": 1,
          "links": [],
          "nullPointMode": "null",
          "percentage": false,
          "pointradius": 5,
          "points": false,
          "renderer": "flot",
          "seriesOverrides": [
            {
              "alias": "/.*_count_.*/",
              "yaxis": 2
            }
          ],
          "spaceLength": 10,
          "stack": false,
          "steppedLine": false,
          "targets": [
            {
              "expr": "[[app]]_fuseclient_OpMetaTruncate{app=~\"$app\", cluster=\"$cluster\", instance=\"$instance\"}",
              "format": "time_series",
              "hide": false,
              "intervalFactor": 1,
              "legendFormat": "Truncate_{{instance}}",
              "refId": "B"
            },
            {
              "expr": "rate([[app]]_fuseclient_OpMetaTruncate_count{app=~\"$app\", cluster=\"$cluster\", instance=\"$instance\"}[1m])",
              "format": "time_series",
              "hide": false,
              "intervalFactor": 1,
              "legendFormat": "Truncate_count_{{instance}}",
              "refId": "L"
            }
          ],
          "thresholds": [],
          "timeFrom": null,
          "timeShift": null,
          "title": "Truncate",
          "tooltip": {
            "shared": true,
            "sort": 0,
            "value_type": "individual"
          },
          "type": "graph",
          "xaxis": {
            "buckets": null,
            "mode": "time",
            "name": null,
            "show": true,
            "values": []
          },
          "yaxes": [
            {
              "format": "ns",
              "label": null,
              "logBase": 1,
              "max": null,
              "min": null,
              "show": true
            },
            {
              "format": "ops",
              "label": null,
              "logBase": 1,
              "max": null,
              "min": null,
              "show": true
            }
          ],
          "yaxis": {
            "align": false,
            "alignLevel": null
          }
        },
        {
          "aliasColors": {},
          "bars": false,
          "dashLength": 10,
          "dashes": false,
          "datasource": null,
          "fill": 1,
          "gridPos": {
            "h": 7,
            "w": 8,
            "x": 16,
            "y": 54
          },
          "id": 151,
          "legend": {
            "avg": false,
            "current": false,
            "max": false,
            "min": false,
            "show": true,
            "total": false,
            "values": false
          },
          "lines": true,
          "linewidth": 1,
          "links": [],
          "nullPointMode": "null",
          "percentage": false,
          "pointradius": 5,
          "points": false,
          "renderer": "flot",
          "seriesOverrides": [],
          "spaceLength": 10,
          "stack": false,
          "steppedLine": false,
          "targets": [
            {
              "expr": "[[app]]_fuseclient_FlushInRead{app=\"$app\", cluster=\"$cluster\", instance=\"$instance\"}",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "FlushInRead_{{instance}}",
              "refId": "A"
            },
            {
              "expr": "[[app]]_fuseclient_PrepareReadRequest{app=~\"$app\", cluster=\"$cluster\", instance=\"$instance\"}",
              "format": "time_series",
              "hide": false,
              "intervalFactor": 1,
              "legendFormat": "PrepareReadRequest_{{instance}}",
              "refId": "B"
            }
          ],
          "thresholds": [],
          "timeFrom": null,
          "timeShift": null,
          "title": "PrepareReadRequest",
          "tooltip": {
            "shared": true,
            "sort": 0,
            "value_type": "individual"
          },
          "type": "graph",
          "xaxis": {
            "buckets": null,
            "mode": "time",
            "name": null,
            "show": true,
            "values": []
          },
          "yaxes": [
            {
              "format": "ns",
              "label": null,
              "logBase": 1,
              "max": null,
              "min": null,
              "show": true
            },
            {
              "format": "short",
              "label": null,
              "logBase": 1,
              "max": null,
              "min": null,
              "show": true
            }
          ],
          "yaxis": {
            "align": false,
            "alignLevel": null
          }
        }
      ],
      "title": "FuseClient",
      "type": "row"
    },
    {
      "collapsed": true,
      "gridPos": {
        "h": 1,
        "w": 24,
        "x": 0,
        "y": 53
      },
      "id": 60,
      "panels": [
        {
          "aliasColors": {},
          "bars": false,
          "dashLength": 10,
          "dashes": false,
          "datasource": null,
          "fill": 1,
          "gridPos": {
            "h": 6,
            "w": 7,
            "x": 0,
            "y": 2
          },
          "id": 61,
          "legend": {
            "avg": false,
            "current": false,
            "max": false,
            "min": false,
            "show": true,
            "total": false,
            "values": false
          },
          "lines": true,
          "linewidth": 1,
          "links": [],
          "nullPointMode": "null",
          "percentage": false,
          "pointradius": 5,
          "points": false,
          "renderer": "flot",
          "seriesOverrides": [],
          "spaceLength": 10,
          "stack": false,
          "steppedLine": false,
          "targets": [
            {
              "expr": "go_goroutines{app=\"$app\", cluster=\"$cluster\", role=\"$role\"}",
              "format": "time_series",
              "intervalFactor": 1,
              "legendFormat": "go_goroutines_{{instance}}",
              "refId": "A"
            },
            {
              "expr": "go_threads{app=\"$app\", cluster=\"$cluster\", role=\"$role\"}",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "go_threads_{{instance}}",
              "refId": "B"
            }
          ],
          "thresholds": [],
          "timeFrom": null,
          "timeShift": null,
          "title": "go_goroutines",
          "tooltip": {
            "shared": true,
            "sort": 0,
            "value_type": "individual"
          },
          "type": "graph",
          "xaxis": {
            "buckets": null,
            "mode": "time",
            "name": null,
            "show": true,
            "values": []
          },
          "yaxes": [
            {
              "format": "locale",
              "label": null,
              "logBase": 1,
              "max": null,
              "min": null,
              "show": true
            },
            {
              "format": "short",
              "label": null,
              "logBase": 1,
              "max": null,
              "min": null,
              "show": true
            }
          ],
          "yaxis": {
            "align": false,
            "alignLevel": null
          }
        },
        {
          "aliasColors": {},
          "bars": false,
          "dashLength": 10,
          "dashes": false,
          "datasource": null,
          "fill": 1,
          "gridPos": {
            "h": 6,
            "w": 7,
            "x": 7,
            "y": 2
          },
          "id": 169,
          "legend": {
            "avg": false,
            "current": false,
            "max": false,
            "min": false,
            "show": true,
            "total": false,
            "values": false
          },
          "lines": true,
          "linewidth": 1,
          "links": [],
          "nullPointMode": "null",
          "percentage": false,
          "pointradius": 5,
          "points": false,
          "renderer": "flot",
          "seriesOverrides": [],
          "spaceLength": 10,
          "stack": false,
          "steppedLine": false,
          "targets": [
            {
              "expr": "go_goroutines{app=\"$app\", cluster=\"$cluster\", role=\"$role\"}",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "go_goroutines_{{instance}}",
              "refId": "A"
            },
            {
              "expr": "go_threads{app=\"$app\", cluster=\"$cluster\", role=\"$role\"}",
              "format": "time_series",
              "hide": false,
              "intervalFactor": 1,
              "legendFormat": "go_threads_{{instance}}",
              "refId": "B"
            }
          ],
          "thresholds": [],
          "timeFrom": null,
          "timeShift": null,
          "title": "go_threads",
          "tooltip": {
            "shared": true,
            "sort": 0,
            "value_type": "individual"
          },
          "type": "graph",
          "xaxis": {
            "buckets": null,
            "mode": "time",
            "name": null,
            "show": true,
            "values": []
          },
          "yaxes": [
            {
              "format": "locale",
              "label": null,
              "logBase": 1,
              "max": null,
              "min": null,
              "show": true
            },
            {
              "format": "short",
              "label": null,
              "logBase": 1,
              "max": null,
              "min": null,
              "show": true
            }
          ],
          "yaxis": {
            "align": false,
            "alignLevel": null
          }
        },
        {
          "aliasColors": {},
          "bars": false,
          "dashLength": 10,
          "dashes": false,
          "datasource": null,
          "fill": 1,
          "gridPos": {
            "h": 6,
            "w": 7,
            "x": 14,
            "y": 2
          },
          "id": 63,
          "legend": {
            "avg": false,
            "current": false,
            "max": false,
            "min": false,
            "show": true,
            "total": false,
            "values": false
          },
          "lines": true,
          "linewidth": 1,
          "links": [],
          "nullPointMode": "null",
          "percentage": false,
          "pointradius": 5,
          "points": false,
          "renderer": "flot",
          "seriesOverrides": [
            {
              "alias": "gc_rate",
              "yaxis": 2
            }
          ],
          "spaceLength": 10,
          "stack": false,
          "steppedLine": false,
          "targets": [
            {
              "expr": "go_gc_duration_seconds{instance=~\"$instance\"}",
              "format": "time_series",
              "intervalFactor": 1,
              "legendFormat": "seconds_{{quantile}}",
              "refId": "A"
            },
            {
              "expr": "rate(go_gc_duration_seconds_count{instance=~\"$instance\"}[1m])",
              "format": "time_series",
              "intervalFactor": 1,
              "legendFormat": "gc_rate",
              "refId": "B"
            }
          ],
          "thresholds": [],
          "timeFrom": null,
          "timeShift": null,
          "title": "gc_duration",
          "tooltip": {
            "shared": true,
            "sort": 0,
            "value_type": "individual"
          },
          "type": "graph",
          "xaxis": {
            "buckets": null,
            "mode": "time",
            "name": null,
            "show": true,
            "values": []
          },
          "yaxes": [
            {
              "format": "s",
              "label": null,
              "logBase": 1,
              "max": null,
              "min": null,
              "show": true
            },
            {
              "format": "locale",
              "label": null,
              "logBase": 1,
              "max": null,
              "min": null,
              "show": true
            }
          ],
          "yaxis": {
            "align": false,
            "alignLevel": null
          }
        },
        {
          "aliasColors": {},
          "bars": false,
          "dashLength": 10,
          "dashes": false,
          "datasource": null,
          "fill": 1,
          "gridPos": {
            "h": 6,
            "w": 7,
            "x": 0,
            "y": 8
          },
          "id": 109,
          "legend": {
            "avg": false,
            "current": false,
            "max": false,
            "min": false,
            "show": true,
            "total": false,
            "values": false
          },
          "lines": true,
          "linewidth": 1,
          "links": [],
          "nullPointMode": "null",
          "percentage": false,
          "pointradius": 5,
          "points": false,
          "renderer": "flot",
          "seriesOverrides": [
            {
              "alias": "gc_rate",
              "yaxis": 2
            }
          ],
          "spaceLength": 10,
          "stack": false,
          "steppedLine": false,
          "targets": [
            {
              "expr": "process_resident_memory_bytes{app=\"$app\", cluster=\"$cluster\", role=\"$role\" }",
              "format": "time_series",
              "hide": false,
              "intervalFactor": 1,
              "legendFormat": "resident_memory_bytes_{{instance}}",
              "refId": "A"
            },
            {
              "expr": "process_virtual_memory_bytes{app=\"$app\", cluster=\"$cluster\", role=\"$role\"}",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "virtual_memory_bytes_{{instance}}",
              "refId": "B"
            },
            {
              "expr": "process_virtual_memory_max_bytes{app=\"$app\", cluster=\"$cluster\", role=\"$role\"}",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "virtual_memory_max_bytes_{{instance}}",
              "refId": "C"
            }
          ],
          "thresholds": [],
          "timeFrom": null,
          "timeShift": null,
          "title": "process_resident_memory",
          "tooltip": {
            "shared": true,
            "sort": 0,
            "value_type": "individual"
          },
          "type": "graph",
          "xaxis": {
            "buckets": null,
            "mode": "time",
            "name": null,
            "show": true,
            "values": []
          },
          "yaxes": [
            {
              "format": "decbytes",
              "label": null,
              "logBase": 1,
              "max": null,
              "min": null,
              "show": true
            },
            {
              "format": "locale",
              "label": null,
              "logBase": 1,
              "max": null,
              "min": null,
              "show": true
            }
          ],
          "yaxis": {
            "align": false,
            "alignLevel": null
          }
        },
        {
          "aliasColors": {},
          "bars": false,
          "dashLength": 10,
          "dashes": false,
          "datasource": null,
          "fill": 1,
          "gridPos": {
            "h": 6,
            "w": 7,
            "x": 7,
            "y": 8
          },
          "id": 166,
          "legend": {
            "avg": false,
            "current": false,
            "max": false,
            "min": false,
            "show": true,
            "total": false,
            "values": false
          },
          "lines": true,
          "linewidth": 1,
          "links": [],
          "nullPointMode": "null",
          "percentage": false,
          "pointradius": 5,
          "points": false,
          "renderer": "flot",
          "seriesOverrides": [
            {
              "alias": "gc_rate",
              "yaxis": 2
            }
          ],
          "spaceLength": 10,
          "stack": false,
          "steppedLine": false,
          "targets": [
            {
              "expr": "process_resident_memory_bytes{app=\"$app\", cluster=\"$cluster\", role=\"$role\" }",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "resident_memory_bytes_{{instance}}",
              "refId": "A"
            },
            {
              "expr": "process_virtual_memory_bytes{app=\"$app\", cluster=\"$cluster\", role=\"$role\"}",
              "format": "time_series",
              "hide": false,
              "intervalFactor": 1,
              "legendFormat": "virtual_memory_bytes_{{instance}}",
              "refId": "B"
            },
            {
              "expr": "process_virtual_memory_max_bytes{app=\"$app\", cluster=\"$cluster\", role=\"$role\"}",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "virtual_memory_max_bytes_{{instance}}",
              "refId": "C"
            }
          ],
          "thresholds": [],
          "timeFrom": null,
          "timeShift": null,
          "title": "process_virtual_memory",
          "tooltip": {
            "shared": true,
            "sort": 0,
            "value_type": "individual"
          },
          "type": "graph",
          "xaxis": {
            "buckets": null,
            "mode": "time",
            "name": null,
            "show": true,
            "values": []
          },
          "yaxes": [
            {
              "format": "decbytes",
              "label": null,
              "logBase": 1,
              "max": null,
              "min": null,
              "show": true
            },
            {
              "format": "locale",
              "label": null,
              "logBase": 1,
              "max": null,
              "min": null,
              "show": true
            }
          ],
          "yaxis": {
            "align": false,
            "alignLevel": null
          }
        },
        {
          "aliasColors": {},
          "bars": false,
          "dashLength": 10,
          "dashes": false,
          "datasource": null,
          "fill": 1,
          "gridPos": {
            "h": 6,
            "w": 7,
            "x": 14,
            "y": 8
          },
          "id": 167,
          "legend": {
            "avg": false,
            "current": false,
            "max": false,
            "min": false,
            "show": true,
            "total": false,
            "values": false
          },
          "lines": true,
          "linewidth": 1,
          "links": [],
          "nullPointMode": "null",
          "percentage": false,
          "pointradius": 5,
          "points": false,
          "renderer": "flot",
          "seriesOverrides": [
            {
              "alias": "gc_rate",
              "yaxis": 2
            }
          ],
          "spaceLength": 10,
          "stack": false,
          "steppedLine": false,
          "targets": [
            {
              "expr": "process_resident_memory_bytes{app=\"$app\", cluster=\"$cluster\", role=\"$role\" }",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "resident_memory_bytes_{{instance}}",
              "refId": "A"
            },
            {
              "expr": "process_virtual_memory_bytes{app=\"$app\", cluster=\"$cluster\", role=\"$role\"}",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "virtual_memory_bytes_{{instance}}",
              "refId": "B"
            },
            {
              "expr": "process_virtual_memory_max_bytes{app=\"$app\", cluster=\"$cluster\", role=\"$role\"}",
              "format": "time_series",
              "hide": false,
              "intervalFactor": 1,
              "legendFormat": "virtual_memory_max_bytes_{{instance}}",
              "refId": "C"
            }
          ],
          "thresholds": [],
          "timeFrom": null,
          "timeShift": null,
          "title": "process_virtual_memory_max",
          "tooltip": {
            "shared": true,
            "sort": 0,
            "value_type": "individual"
          },
          "type": "graph",
          "xaxis": {
            "buckets": null,
            "mode": "time",
            "name": null,
            "show": true,
            "values": []
          },
          "yaxes": [
            {
              "format": "decbytes",
              "label": null,
              "logBase": 1,
              "max": null,
              "min": null,
              "show": true
            },
            {
              "format": "locale",
              "label": null,
              "logBase": 1,
              "max": null,
              "min": null,
              "show": true
            }
          ],
          "yaxis": {
            "align": false,
            "alignLevel": null
          }
        },
        {
          "aliasColors": {},
          "bars": false,
          "dashLength": 10,
          "dashes": false,
          "datasource": "default",
          "fill": 1,
          "gridPos": {
            "h": 6,
            "w": 7,
            "x": 0,
            "y": 14
          },
          "id": 110,
          "legend": {
            "avg": false,
            "current": false,
            "max": false,
            "min": false,
            "show": true,
            "total": false,
            "values": false
          },
          "lines": true,
          "linewidth": 1,
          "links": [],
          "nullPointMode": "null",
          "percentage": false,
          "pointradius": 5,
          "points": false,
          "renderer": "flot",
          "seriesOverrides": [
            {
              "alias": "gc_rate",
              "yaxis": 2
            }
          ],
          "spaceLength": 10,
          "stack": false,
          "steppedLine": false,
          "targets": [
            {
              "expr": "process_open_fds{app=\"$app\", cluster=\"$cluster\", role=\"$role\"}",
              "format": "time_series",
              "intervalFactor": 1,
              "legendFormat": "open_fds_{{instance}}",
              "refId": "A"
            },
            {
              "expr": "process_max_fds{app=\"$app\", cluster=\"$cluster\", role=\"$role\"}",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "max_fds_{{instance}}",
              "refId": "B"
            }
          ],
          "thresholds": [],
          "timeFrom": null,
          "timeShift": null,
          "title": "open_fds",
          "tooltip": {
            "shared": true,
            "sort": 0,
            "value_type": "individual"
          },
          "type": "graph",
          "xaxis": {
            "buckets": null,
            "mode": "time",
            "name": null,
            "show": true,
            "values": []
          },
          "yaxes": [
            {
              "format": "locale",
              "label": null,
              "logBase": 1,
              "max": null,
              "min": null,
              "show": true
            },
            {
              "format": "locale",
              "label": null,
              "logBase": 1,
              "max": null,
              "min": null,
              "show": true
            }
          ],
          "yaxis": {
            "align": false,
            "alignLevel": null
          }
        },
        {
          "aliasColors": {},
          "bars": false,
          "dashLength": 10,
          "dashes": false,
          "datasource": "default",
          "fill": 1,
          "gridPos": {
            "h": 6,
            "w": 7,
            "x": 7,
            "y": 14
          },
          "id": 168,
          "legend": {
            "avg": false,
            "current": false,
            "max": false,
            "min": false,
            "show": true,
            "total": false,
            "values": false
          },
          "lines": true,
          "linewidth": 1,
          "links": [],
          "nullPointMode": "null",
          "percentage": false,
          "pointradius": 5,
          "points": false,
          "renderer": "flot",
          "seriesOverrides": [
            {
              "alias": "gc_rate",
              "yaxis": 2
            }
          ],
          "spaceLength": 10,
          "stack": false,
          "steppedLine": false,
          "targets": [
            {
              "expr": "process_open_fds{app=\"$app\", cluster=\"$cluster\", role=\"$role\"}",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "open_fds_{{instance}}",
              "refId": "A"
            },
            {
              "expr": "process_max_fds{app=\"$app\", cluster=\"$cluster\", role=\"$role\"}",
              "format": "time_series",
              "hide": false,
              "intervalFactor": 1,
              "legendFormat": "max_fds_{{instance}}",
              "refId": "B"
            }
          ],
          "thresholds": [],
          "timeFrom": null,
          "timeShift": null,
          "title": "max_fds",
          "tooltip": {
            "shared": true,
            "sort": 0,
            "value_type": "individual"
          },
          "type": "graph",
          "xaxis": {
            "buckets": null,
            "mode": "time",
            "name": null,
            "show": true,
            "values": []
          },
          "yaxes": [
            {
              "format": "locale",
              "label": null,
              "logBase": 1,
              "max": null,
              "min": null,
              "show": true
            },
            {
              "format": "locale",
              "label": null,
              "logBase": 1,
              "max": null,
              "min": null,
              "show": true
            }
          ],
          "yaxis": {
            "align": false,
            "alignLevel": null
          }
        },
        {
          "aliasColors": {},
          "bars": false,
          "dashLength": 10,
          "dashes": false,
          "datasource": null,
          "fill": 1,
          "gridPos": {
            "h": 6,
            "w": 7,
            "x": 14,
            "y": 14
          },
          "id": 111,
          "legend": {
            "avg": false,
            "current": false,
            "max": false,
            "min": false,
            "show": true,
            "total": false,
            "values": false
          },
          "lines": true,
          "linewidth": 1,
          "links": [],
          "nullPointMode": "null",
          "percentage": false,
          "pointradius": 5,
          "points": false,
          "renderer": "flot",
          "seriesOverrides": [
            {
              "alias": "gc_rate",
              "yaxis": 2
            }
          ],
          "spaceLength": 10,
          "stack": false,
          "steppedLine": false,
          "targets": [
            {
              "expr": "rate(process_cpu_seconds_total{cluster=\"$cluster\", role=\"$role\"}[1m])",
              "format": "time_series",
              "intervalFactor": 1,
              "legendFormat": "cpu_seconds_{{instance}}",
              "refId": "A"
            }
          ],
          "thresholds": [],
          "timeFrom": null,
          "timeShift": null,
          "title": "process_cpu",
          "tooltip": {
            "shared": true,
            "sort": 0,
            "value_type": "individual"
          },
          "type": "graph",
          "xaxis": {
            "buckets": null,
            "mode": "time",
            "name": null,
            "show": true,
            "values": []
          },
          "yaxes": [
            {
              "format": "s",
              "label": null,
              "logBase": 1,
              "max": null,
              "min": null,
              "show": true
            },
            {
              "format": "locale",
              "label": null,
              "logBase": 1,
              "max": null,
              "min": null,
              "show": true
            }
          ],
          "yaxis": {
            "align": false,
            "alignLevel": null
          }
        },
        {
          "aliasColors": {},
          "bars": false,
          "dashLength": 10,
          "dashes": false,
          "datasource": null,
          "fill": 1,
          "gridPos": {
            "h": 6,
            "w": 7,
            "x": 0,
            "y": 20
          },
          "id": 62,
          "legend": {
            "avg": false,
            "current": false,
            "max": false,
            "min": false,
            "show": true,
            "total": false,
            "values": false
          },
          "lines": true,
          "linewidth": 1,
          "links": [],
          "nullPointMode": "null",
          "percentage": false,
          "pointradius": 5,
          "points": false,
          "renderer": "flot",
          "seriesOverrides": [],
          "spaceLength": 10,
          "stack": false,
          "steppedLine": false,
          "targets": [
            {
              "expr": "go_memstats_alloc_bytes{app=\"$app\", cluster=\"$cluster\", role=\"$role\"}",
              "format": "time_series",
              "intervalFactor": 1,
              "legendFormat": "alloc_bytes_{{instance}}",
              "refId": "A"
            },
            {
              "expr": "go_memstats_alloc_bytes_total{app=\"$app\", cluster=\"$cluster\", role=\"$role\"}",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "alloc_bytes_total_{{instance}}",
              "refId": "B"
            },
            {
              "expr": "go_memstats_heap_alloc_bytes{app=\"$app\", cluster=\"$cluster\", role=\"$role\"}",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "heap_alloc_bytes_{{instance}}",
              "refId": "C"
            },
            {
              "expr": "go_memstats_heap_inuse_bytes{app=\"$app\", cluster=\"$cluster\", role=\"$role\"}",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "heap_inuse_bytes_{{instance}}",
              "refId": "D"
            },
            {
              "expr": "go_memstats_sys_bytes{app=\"$app\", cluster=\"$cluster\", role=\"$role\"}",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "sys_bytes_{{instance}}",
              "refId": "E"
            },
            {
              "expr": "go_memstats_mallocs_total{app=\"$app\", cluster=\"$cluster\", role=\"$role\"}",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "mallocs_total_{{instance}}",
              "refId": "F"
            },
            {
              "expr": "go_memstats_frees_total{app=\"$app\", cluster=\"$cluster\", role=\"$role\"}",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "frees_total_{{instance}}",
              "refId": "G"
            },
            {
              "expr": "go_memstats_other_sys_bytes{app=\"$app\", cluster=\"$cluster\", role=\"$role\"}",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "other_sys_bytes_{{instance}}",
              "refId": "H"
            },
            {
              "expr": "go_memstats_mcache_sys_bytes{app=\"$app\", cluster=\"$cluster\", role=\"$role\"}",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "mcache_sys_bytes_{{instance}}",
              "refId": "I"
            }
          ],
          "thresholds": [],
          "timeFrom": null,
          "timeShift": null,
          "title": "alloc_bytes",
          "tooltip": {
            "shared": true,
            "sort": 0,
            "value_type": "individual"
          },
          "type": "graph",
          "xaxis": {
            "buckets": null,
            "mode": "time",
            "name": null,
            "show": true,
            "values": []
          },
          "yaxes": [
            {
              "format": "decbytes",
              "label": null,
              "logBase": 1,
              "max": null,
              "min": null,
              "show": true
            },
            {
              "format": "short",
              "label": null,
              "logBase": 1,
              "max": null,
              "min": null,
              "show": true
            }
          ],
          "yaxis": {
            "align": false,
            "alignLevel": null
          }
        },
        {
          "aliasColors": {},
          "bars": false,
          "dashLength": 10,
          "dashes": false,
          "datasource": null,
          "fill": 1,
          "gridPos": {
            "h": 6,
            "w": 7,
            "x": 7,
            "y": 20
          },
          "id": 172,
          "legend": {
            "avg": false,
            "current": false,
            "max": false,
            "min": false,
            "show": true,
            "total": false,
            "values": false
          },
          "lines": true,
          "linewidth": 1,
          "links": [],
          "nullPointMode": "null",
          "percentage": false,
          "pointradius": 5,
          "points": false,
          "renderer": "flot",
          "seriesOverrides": [],
          "spaceLength": 10,
          "stack": false,
          "steppedLine": false,
          "targets": [
            {
              "expr": "go_memstats_alloc_bytes{app=\"$app\", cluster=\"$cluster\", role=\"$role\"}",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "alloc_bytes_{{instance}}",
              "refId": "A"
            },
            {
              "expr": "go_memstats_alloc_bytes_total{app=\"$app\", cluster=\"$cluster\", role=\"$role\"}",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "alloc_bytes_total_{{instance}}",
              "refId": "B"
            },
            {
              "expr": "go_memstats_heap_alloc_bytes{app=\"$app\", cluster=\"$cluster\", role=\"$role\"}",
              "format": "time_series",
              "hide": false,
              "intervalFactor": 1,
              "legendFormat": "heap_alloc_bytes_{{instance}}",
              "refId": "C"
            },
            {
              "expr": "go_memstats_heap_inuse_bytes{app=\"$app\", cluster=\"$cluster\", role=\"$role\"}",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "heap_inuse_bytes_{{instance}}",
              "refId": "D"
            },
            {
              "expr": "go_memstats_sys_bytes{app=\"$app\", cluster=\"$cluster\", role=\"$role\"}",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "sys_bytes_{{instance}}",
              "refId": "E"
            },
            {
              "expr": "go_memstats_mallocs_total{app=\"$app\", cluster=\"$cluster\", role=\"$role\"}",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "mallocs_total_{{instance}}",
              "refId": "F"
            },
            {
              "expr": "go_memstats_frees_total{app=\"$app\", cluster=\"$cluster\", role=\"$role\"}",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "frees_total_{{instance}}",
              "refId": "G"
            },
            {
              "expr": "go_memstats_other_sys_bytes{app=\"$app\", cluster=\"$cluster\", role=\"$role\"}",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "other_sys_bytes_{{instance}}",
              "refId": "H"
            },
            {
              "expr": "go_memstats_mcache_sys_bytes{app=\"$app\", cluster=\"$cluster\", role=\"$role\"}",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "mcache_sys_bytes_{{instance}}",
              "refId": "I"
            }
          ],
          "thresholds": [],
          "timeFrom": null,
          "timeShift": null,
          "title": "heap_alloc",
          "tooltip": {
            "shared": true,
            "sort": 0,
            "value_type": "individual"
          },
          "type": "graph",
          "xaxis": {
            "buckets": null,
            "mode": "time",
            "name": null,
            "show": true,
            "values": []
          },
          "yaxes": [
            {
              "format": "decbytes",
              "label": null,
              "logBase": 1,
              "max": null,
              "min": null,
              "show": true
            },
            {
              "format": "short",
              "label": null,
              "logBase": 1,
              "max": null,
              "min": null,
              "show": true
            }
          ],
          "yaxis": {
            "align": false,
            "alignLevel": null
          }
        },
        {
          "aliasColors": {},
          "bars": false,
          "dashLength": 10,
          "dashes": false,
          "datasource": null,
          "fill": 1,
          "gridPos": {
            "h": 6,
            "w": 7,
            "x": 14,
            "y": 20
          },
          "id": 171,
          "legend": {
            "avg": false,
            "current": false,
            "max": false,
            "min": false,
            "show": true,
            "total": false,
            "values": false
          },
          "lines": true,
          "linewidth": 1,
          "links": [],
          "nullPointMode": "null",
          "percentage": false,
          "pointradius": 5,
          "points": false,
          "renderer": "flot",
          "seriesOverrides": [],
          "spaceLength": 10,
          "stack": false,
          "steppedLine": false,
          "targets": [
            {
              "expr": "go_memstats_alloc_bytes{app=\"$app\", cluster=\"$cluster\", role=\"$role\"}",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "alloc_bytes_{{instance}}",
              "refId": "A"
            },
            {
              "expr": "go_memstats_alloc_bytes_total{app=\"$app\", cluster=\"$cluster\", role=\"$role\"}",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "alloc_bytes_total_{{instance}}",
              "refId": "B"
            },
            {
              "expr": "go_memstats_heap_alloc_bytes{app=\"$app\", cluster=\"$cluster\", role=\"$role\"}",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "heap_alloc_bytes_{{instance}}",
              "refId": "C"
            },
            {
              "expr": "go_memstats_heap_inuse_bytes{app=\"$app\", cluster=\"$cluster\", role=\"$role\"}",
              "format": "time_series",
              "hide": false,
              "intervalFactor": 1,
              "legendFormat": "heap_inuse_bytes_{{instance}}",
              "refId": "D"
            },
            {
              "expr": "go_memstats_sys_bytes{app=\"$app\", cluster=\"$cluster\", role=\"$role\"}",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "sys_bytes_{{instance}}",
              "refId": "E"
            },
            {
              "expr": "go_memstats_mallocs_total{app=\"$app\", cluster=\"$cluster\", role=\"$role\"}",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "mallocs_total_{{instance}}",
              "refId": "F"
            },
            {
              "expr": "go_memstats_frees_total{app=\"$app\", cluster=\"$cluster\", role=\"$role\"}",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "frees_total_{{instance}}",
              "refId": "G"
            },
            {
              "expr": "go_memstats_other_sys_bytes{app=\"$app\", cluster=\"$cluster\", role=\"$role\"}",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "other_sys_bytes_{{instance}}",
              "refId": "H"
            },
            {
              "expr": "go_memstats_mcache_sys_bytes{app=\"$app\", cluster=\"$cluster\", role=\"$role\"}",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "mcache_sys_bytes_{{instance}}",
              "refId": "I"
            }
          ],
          "thresholds": [],
          "timeFrom": null,
          "timeShift": null,
          "title": "heap_inuse",
          "tooltip": {
            "shared": true,
            "sort": 0,
            "value_type": "individual"
          },
          "type": "graph",
          "xaxis": {
            "buckets": null,
            "mode": "time",
            "name": null,
            "show": true,
            "values": []
          },
          "yaxes": [
            {
              "format": "decbytes",
              "label": null,
              "logBase": 1,
              "max": null,
              "min": null,
              "show": true
            },
            {
              "format": "short",
              "label": null,
              "logBase": 1,
              "max": null,
              "min": null,
              "show": true
            }
          ],
          "yaxis": {
            "align": false,
            "alignLevel": null
          }
        },
        {
          "aliasColors": {},
          "bars": false,
          "dashLength": 10,
          "dashes": false,
          "datasource": null,
          "fill": 1,
          "gridPos": {
            "h": 6,
            "w": 7,
            "x": 0,
            "y": 26
          },
          "id": 170,
          "legend": {
            "avg": false,
            "current": false,
            "max": false,
            "min": false,
            "show": true,
            "total": false,
            "values": false
          },
          "lines": true,
          "linewidth": 1,
          "links": [],
          "nullPointMode": "null",
          "percentage": false,
          "pointradius": 5,
          "points": false,
          "renderer": "flot",
          "seriesOverrides": [],
          "spaceLength": 10,
          "stack": false,
          "steppedLine": false,
          "targets": [
            {
              "expr": "go_memstats_alloc_bytes{app=\"$app\", cluster=\"$cluster\", role=\"$role\"}",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "alloc_bytes_{{instance}}",
              "refId": "A"
            },
            {
              "expr": "go_memstats_alloc_bytes_total{app=\"$app\", cluster=\"$cluster\", role=\"$role\"}",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "alloc_bytes_total_{{instance}}",
              "refId": "B"
            },
            {
              "expr": "go_memstats_heap_alloc_bytes{app=\"$app\", cluster=\"$cluster\", role=\"$role\"}",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "heap_alloc_bytes_{{instance}}",
              "refId": "C"
            },
            {
              "expr": "go_memstats_heap_inuse_bytes{app=\"$app\", cluster=\"$cluster\", role=\"$role\"}",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "heap_inuse_bytes_{{instance}}",
              "refId": "D"
            },
            {
              "expr": "go_memstats_sys_bytes{app=\"$app\", cluster=\"$cluster\", role=\"$role\"}",
              "format": "time_series",
              "hide": false,
              "intervalFactor": 1,
              "legendFormat": "sys_bytes_{{instance}}",
              "refId": "E"
            },
            {
              "expr": "go_memstats_mallocs_total{app=\"$app\", cluster=\"$cluster\", role=\"$role\"}",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "mallocs_total_{{instance}}",
              "refId": "F"
            },
            {
              "expr": "go_memstats_frees_total{app=\"$app\", cluster=\"$cluster\", role=\"$role\"}",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "frees_total_{{instance}}",
              "refId": "G"
            },
            {
              "expr": "go_memstats_other_sys_bytes{app=\"$app\", cluster=\"$cluster\", role=\"$role\"}",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "other_sys_bytes_{{instance}}",
              "refId": "H"
            },
            {
              "expr": "go_memstats_mcache_sys_bytes{app=\"$app\", cluster=\"$cluster\", role=\"$role\"}",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "mcache_sys_bytes_{{instance}}",
              "refId": "I"
            }
          ],
          "thresholds": [],
          "timeFrom": null,
          "timeShift": null,
          "title": "sys_bytes",
          "tooltip": {
            "shared": true,
            "sort": 0,
            "value_type": "individual"
          },
          "type": "graph",
          "xaxis": {
            "buckets": null,
            "mode": "time",
            "name": null,
            "show": true,
            "values": []
          },
          "yaxes": [
            {
              "format": "decbytes",
              "label": null,
              "logBase": 1,
              "max": null,
              "min": null,
              "show": true
            },
            {
              "format": "short",
              "label": null,
              "logBase": 1,
              "max": null,
              "min": null,
              "show": true
            }
          ],
          "yaxis": {
            "align": false,
            "alignLevel": null
          }
        },
        {
          "aliasColors": {},
          "bars": false,
          "dashLength": 10,
          "dashes": false,
          "datasource": null,
          "fill": 1,
          "gridPos": {
            "h": 6,
            "w": 7,
            "x": 7,
            "y": 26
          },
          "id": 173,
          "legend": {
            "avg": false,
            "current": false,
            "max": false,
            "min": false,
            "show": true,
            "total": false,
            "values": false
          },
          "lines": true,
          "linewidth": 1,
          "links": [],
          "nullPointMode": "null",
          "percentage": false,
          "pointradius": 5,
          "points": false,
          "renderer": "flot",
          "seriesOverrides": [],
          "spaceLength": 10,
          "stack": false,
          "steppedLine": false,
          "targets": [
            {
              "expr": "go_memstats_alloc_bytes{app=\"$app\", cluster=\"$cluster\", role=\"$role\"}",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "alloc_bytes_{{instance}}",
              "refId": "A"
            },
            {
              "expr": "go_memstats_alloc_bytes_total{app=\"$app\", cluster=\"$cluster\", role=\"$role\"}",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "alloc_bytes_total_{{instance}}",
              "refId": "B"
            },
            {
              "expr": "go_memstats_heap_alloc_bytes{app=\"$app\", cluster=\"$cluster\", role=\"$role\"}",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "heap_alloc_bytes_{{instance}}",
              "refId": "C"
            },
            {
              "expr": "go_memstats_heap_inuse_bytes{app=\"$app\", cluster=\"$cluster\", role=\"$role\"}",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "heap_inuse_bytes_{{instance}}",
              "refId": "D"
            },
            {
              "expr": "go_memstats_sys_bytes{app=\"$app\", cluster=\"$cluster\", role=\"$role\"}",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "sys_bytes_{{instance}}",
              "refId": "E"
            },
            {
              "expr": "go_memstats_mallocs_total{app=\"$app\", cluster=\"$cluster\", role=\"$role\"}",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "mallocs_total_{{instance}}",
              "refId": "F"
            },
            {
              "expr": "go_memstats_frees_total{app=\"$app\", cluster=\"$cluster\", role=\"$role\"}",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "frees_total_{{instance}}",
              "refId": "G"
            },
            {
              "expr": "go_memstats_other_sys_bytes{app=\"$app\", cluster=\"$cluster\", role=\"$role\"}",
              "format": "time_series",
              "hide": false,
              "intervalFactor": 1,
              "legendFormat": "other_sys_bytes_{{instance}}",
              "refId": "H"
            },
            {
              "expr": "go_memstats_mcache_sys_bytes{app=\"$app\", cluster=\"$cluster\", role=\"$role\"}",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "mcache_sys_bytes_{{instance}}",
              "refId": "I"
            }
          ],
          "thresholds": [],
          "timeFrom": null,
          "timeShift": null,
          "title": "other_sys",
          "tooltip": {
            "shared": true,
            "sort": 0,
            "value_type": "individual"
          },
          "type": "graph",
          "xaxis": {
            "buckets": null,
            "mode": "time",
            "name": null,
            "show": true,
            "values": []
          },
          "yaxes": [
            {
              "format": "decbytes",
              "label": null,
              "logBase": 1,
              "max": null,
              "min": null,
              "show": true
            },
            {
              "format": "short",
              "label": null,
              "logBase": 1,
              "max": null,
              "min": null,
              "show": true
            }
          ],
          "yaxis": {
            "align": false,
            "alignLevel": null
          }
        },
        {
          "aliasColors": {},
          "bars": false,
          "dashLength": 10,
          "dashes": false,
          "datasource": null,
          "fill": 1,
          "gridPos": {
            "h": 6,
            "w": 7,
            "x": 14,
            "y": 26
          },
          "id": 174,
          "legend": {
            "avg": false,
            "current": false,
            "max": false,
            "min": false,
            "show": true,
            "total": false,
            "values": false
          },
          "lines": true,
          "linewidth": 1,
          "links": [],
          "nullPointMode": "null",
          "percentage": false,
          "pointradius": 5,
          "points": false,
          "renderer": "flot",
          "seriesOverrides": [],
          "spaceLength": 10,
          "stack": false,
          "steppedLine": false,
          "targets": [
            {
              "expr": "go_memstats_alloc_bytes{app=\"$app\", cluster=\"$cluster\", role=\"$role\"}",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "alloc_bytes_{{instance}}",
              "refId": "A"
            },
            {
              "expr": "go_memstats_alloc_bytes_total{app=\"$app\", cluster=\"$cluster\", role=\"$role\"}",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "alloc_bytes_total_{{instance}}",
              "refId": "B"
            },
            {
              "expr": "go_memstats_heap_alloc_bytes{app=\"$app\", cluster=\"$cluster\", role=\"$role\"}",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "heap_alloc_bytes_{{instance}}",
              "refId": "C"
            },
            {
              "expr": "go_memstats_heap_inuse_bytes{app=\"$app\", cluster=\"$cluster\", role=\"$role\"}",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "heap_inuse_bytes_{{instance}}",
              "refId": "D"
            },
            {
              "expr": "go_memstats_sys_bytes{app=\"$app\", cluster=\"$cluster\", role=\"$role\"}",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "sys_bytes_{{instance}}",
              "refId": "E"
            },
            {
              "expr": "go_memstats_mallocs_total{app=\"$app\", cluster=\"$cluster\", role=\"$role\"}",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "mallocs_total_{{instance}}",
              "refId": "F"
            },
            {
              "expr": "go_memstats_frees_total{app=\"$app\", cluster=\"$cluster\", role=\"$role\"}",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "frees_total_{{instance}}",
              "refId": "G"
            },
            {
              "expr": "go_memstats_other_sys_bytes{app=\"$app\", cluster=\"$cluster\", role=\"$role\"}",
              "format": "time_series",
              "hide": true,
              "intervalFactor": 1,
              "legendFormat": "other_sys_bytes_{{instance}}",
              "refId": "H"
            },
            {
              "expr": "go_memstats_mcache_sys_bytes{app=\"$app\", cluster=\"$cluster\", role=\"$role\"}",
              "format": "time_series",
              "hide": false,
              "intervalFactor": 1,
              "legendFormat": "mcache_sys_bytes_{{instance}}",
              "refId": "I"
            }
          ],
          "thresholds": [],
          "timeFrom": null,
          "timeShift": null,
          "title": "mcache_sys",
          "tooltip": {
            "shared": true,
            "sort": 0,
            "value_type": "individual"
          },
          "type": "graph",
          "xaxis": {
            "buckets": null,
            "mode": "time",
            "name": null,
            "show": true,
            "values": []
          },
          "yaxes": [
            {
              "format": "decbytes",
              "label": null,
              "logBase": 1,
              "max": null,
              "min": null,
              "show": true
            },
            {
              "format": "short",
              "label": null,
              "logBase": 1,
              "max": null,
              "min": null,
              "show": true
            }
          ],
          "yaxis": {
            "align": false,
            "alignLevel": null
          }
        }
      ],
      "title": "GoRuntime",
      "type": "row"
    }
  ],
  "refresh": false,
  "schemaVersion": 16,
  "style": "dark",
  "tags": [],
  "templating": {
    "list": [
      {
        "allValue": null,
        "current": {
          "selected": true,
          "text": "cfs",
          "value": "cfs"
        },
        "hide": 2,
        "includeAll": false,
        "label": "App",
        "multi": false,
        "name": "app",
        "options": [
          {
            "selected": true,
            "text": "cfs",
            "value": "cfs"
          }
        ],
        "query": "cfs",
        "type": "custom"
      },
      {
        "allValue": null,
        "current": {
          "tags": [],
          "text": "spark",
          "value": "spark"
        },
        "datasource": "default",
        "hide": 0,
        "includeAll": false,
        "label": "Cluster",
        "multi": false,
        "name": "cluster",
        "options": [],
        "query": "label_values(go_info{app=~\"$app\"}, cluster)",
        "refresh": 1,
        "regex": "",
        "sort": 0,
        "tagValuesQuery": "",
        "tags": [],
        "tagsQuery": "",
        "type": "query",
        "useTags": false
      },
      {
        "allValue": null,
        "current": {
          "text": "fuseclient",
          "value": "fuseclient"
        },
        "datasource": "default",
        "hide": 0,
        "includeAll": false,
        "label": "Role",
        "multi": false,
        "name": "role",
        "options": [],
        "query": "label_values(go_info{app=~\"$app\", cluster=~\"$cluster\"}, role)",
        "refresh": 1,
        "regex": "",
        "sort": 0,
        "tagValuesQuery": "",
        "tags": [],
        "tagsQuery": "",
        "type": "query",
        "useTags": false
      },
      {
        "allValue": null,
        "current": {
          "text": "10.170.6.247:9500",
          "value": "10.170.6.247:9500"
        },
        "datasource": "default",
        "hide": 0,
        "includeAll": false,
        "label": "Instance",
        "multi": false,
        "name": "instance",
        "options": [],
        "query": "label_values(go_info{app=~\"$app\", role=~\"$role\", cluster=~\"$cluster\"}, instance)",
        "refresh": 1,
        "regex": "",
        "sort": 0,
        "tagValuesQuery": "",
        "tags": [],
        "tagsQuery": "",
        "type": "query",
        "useTags": false
      },
      {
        "allValue": null,
        "current": {
          "text": "10.170.6.247",
          "value": "10.170.6.247"
        },
        "datasource": "default",
        "hide": 2,
        "includeAll": false,
        "label": "Host",
        "multi": false,
        "name": "hostip",
        "options": [],
        "query": "label_values(go_info{instance=~\"$instance\", cluster=~\"$cluster\"}, instance)",
        "refresh": 1,
        "regex": "/([^:]+):.*/",
        "sort": 0,
        "tagValuesQuery": "",
        "tags": [],
        "tagsQuery": "",
        "type": "query",
        "useTags": false
      },
      {
        "allValue": null,
        "current": {
          "tags": [],
          "text": "clickhourse",
          "value": "clickhourse"
        },
        "datasource": "default",
        "hide": 0,
        "includeAll": false,
        "label": "Vol",
        "multi": false,
        "name": "vol",
        "options": [],
        "query": "label_values(cfs_master_vol_total_GB{app=\"$app\",cluster=\"$cluster\"}, volName)",
        "refresh": 1,
        "regex": "",
        "sort": 0,
        "tagValuesQuery": "",
        "tags": [],
        "tagsQuery": "",
        "type": "query",
        "useTags": false
      }
    ]
  },
  "time": {
    "from": "now-1h",
    "to": "now"
  },
  "timepicker": {
    "refresh_intervals": [
      "5s",
      "10s",
      "30s",
      "1m",
      "5m",
      "15m",
      "30m",
      "1h",
      "2h",
      "1d"
    ],
    "time_options": [
      "5m",
      "15m",
      "1h",
      "6h",
      "12h",
      "24h",
      "2d",
      "7d",
      "30d"
    ]
  },
  "timezone": "",
  "title": "chubaofs",
  "uid": "J8XJyOmZk",
  "version": 484
}

Tune FUSE Performance

Fetch Linux kernel source code

Download the corresponding src rpm, and use the following commands to install source code.

rpm -i kernel-3.10.0-327.28.3.el7.src.rpm 2>&1 | grep -v exist
cd ~/rpmbuild/SPECS
rpmbuild -bp --target=$(uname -m) kernel.spec

The source code will be installed in ~/rpmbuild/BUILD/

Optimize FUSE linux kernel module

In order to achieve maximum throughput performance, several FUSE kernel parameters have to be modified, such as FUSE_MAX_PAGES_PER_REQ and FUSE_DEFAULT_MAX_BACKGROUND.

Update source code according to the following lines.

/* fs/fuse/fuse_i.h */
#define FUSE_MAX_PAGES_PER_REQ 256

/* fs/fuse/inode.c */
#define FUSE_DEFAULT_MAX_BACKGROUND 32

Build against current running Linux kernel

yum install kernel-devel-3.10.0-327.28.3.el7.x86_64

cd ~/rpmbuild/BUILD/kernel-3.10.0-327.28.3.el7/linux-3.10.0-327.28.3.el7.x86_64/fs/fuse
make -C /lib/modules/`uname -r`/build M=$PWD

Install kernel module

cp fuse.ko /lib/modules/`uname -r`/kernel/fs/fuse

rmmod fuse
depmod -a
modprobe fuse

Run Cluster on Docker

Under the docker directory, a helper tool called run_docker.sh is provided to run ChubaoFS with docker-compose.

To start a minimal ChubaoFS cluster from scratch, note that /data/disk is arbitrary, and make sure there are at least 30G available space.

$ docker/run_docker.sh -r -d /data/disk

If client starts successfully, use mount command in client docker shell to check mount status:

$ mount | grep chubaofs

Open http://127.0.0.1:3000 in browser, login with admin/123456 to view grafana monitor metrics.

Or run server and client seperately by following commands:

$ docker/run_docker.sh -b
$ docker/run_docker.sh -s -d /data/disk
$ docker/run_docker.sh -c
$ docker/run_docker.sh -m

For more usage:

$ docker/run_docker.sh -h

Prometheus and Grafana confg can be found in docker/monitor directory.

Resource Manager (Master) API

Cluster

Overview

curl -v "http://10.196.59.198:17010/admin/getCluster" | python -m json.tool

Display the base information of the cluster, such as the detail of metaNode, dataNode, vol and so on.

response

{
    "Name": "test",
    "LeaderAddr": "10.196.59.198:17010",
    "DisableAutoAlloc": false,
    "Applied": 225,
    "MaxDataPartitionID": 100,
    "MaxMetaNodeID": 3,
    "MaxMetaPartitionID": 1,
    "DataNodeStatInfo": {},
    "MetaNodeStatInfo": {},
    "VolStatInfo": {},
    "BadPartitionIDs": {},
    "BadMetaPartitionIDs": {},
    "MetaNodes": {},
    "DataNodes": {}
}

Freeze

curl -v "http://10.196.59.198:17010/cluster/freeze?enable=true"

If cluster is freezed, the vol never allocates dataPartitions automatically.

Parameters
Parameter Type Description
enable bool if enable is true, the cluster is freezed

Statistics

curl -v "http://10.196.59.198:17010/cluster/stat"

Show cluster space information by zone.

response

{
    "DataNodeStatInfo": {
        "TotalGB": 1,
        "UsedGB": 0,
        "IncreasedGB": -2,
        "UsedRatio": "0.0"
    },
    "MetaNodeStatInfo": {
        "TotalGB": 1,
        "UsedGB": 0,
        "IncreasedGB": -8,
        "UsedRatio": "0.0"
    },
    "ZoneStatInfo": {
        "zone1": {
            "DataNodeStat": {
                "TotalGB": 1,
                "UsedGB": 0,
                "AvailGB": 0,
                "UsedRatio": 0,
                "TotalNodes": 0,
                "WritableNodes": 0
            },
            "MetaNodeStat": {
                "TotalGB": 1,
                "UsedGB": 0,
                "AvailGB": 0,
                "UsedRatio": 0,
                "TotalNodes": 0,
                "WritableNodes": 0
            }
        }
    }
}

Topology

curl -v "http://10.196.59.198:17010/topo/get"

Show cluster topology information by zone.

response

[
    {
        "Name": "zone1",
        "Status": "available",
        "NodeSet": {
            "700": {
                "DataNodeLen": 0,
                "MetaNodeLen": 0,
                "MetaNodes": [],
                "DataNodes": []
            }
        }
    },
    {
        "Name": "zone2",
        "Status": "available",
        "NodeSet": {
            "800": {
                "DataNodeLen": 0,
                "MetaNodeLen": 0,
                "MetaNodes": [],
                "DataNodes": []
            }
        }
    }
]

Update Zone

curl -v "http://10.196.59.198:17010/zone/update?name=zone1&enable=false"

Set the status of the zone to available or unavailable.

Parameters
Parameter Type Description
name string zone name
enable bool if enable is true, the cluster is available

Get Zone

curl -v "http://10.196.59.198:17010/zone/list"

Get name and status of all zones.

response

[
    {
        "Name": "zone1",
        "Status": "available",
        "NodeSet": {}
    },
    {
        "Name": "zone2",
        "Status": "available",
        "NodeSet": {}
    }
]

Get Node Info

curl -v "http://192.168.0.11:17010/admin/getNodeInfo"

Get node info of cluster.

response

{
    "code": 0,
    "msg": "success",
    "data": {
        "batchCount": 0,
        "deleteWorkerSleepMs": 0,
        "markDeleteRate": 0
    }
}

Set Node Info

curl -v "http://192.168.0.11:17010/admin/setNodeInfo?batchCount=100&markDeleteRate=100&deleteWorkerSleepMs=1000"

Set node info of cluster.

Parameters
Parameter Type Description
batchCount uint64 metanode delete batch count
deleteWorkerSleepMs uint64 metanode delete worker sleep time with millisecond. if 0 for no sleep
markDeleteRate uint64 datanode batch markdelete limit rate. if 0 for no infinity limit

Volume

Create

curl -v "http://10.196.59.198:17010/admin/createVol?name=test&capacity=100&owner=cfs&mpCount=3"
Allocate a set of data partition and a meta partition to the user.
Default create 10 data partition and 3 meta partition when create volume.
ChubaoFS uses the Owner parameter as the user ID. When creating a volume, if there is no user named the owner of the volume, a user with the user ID same as Owner will be automatically created; if a user named Owner already exists in the cluster, the volume will be owned by the user. For details, please see:: doc: /admin-api/master/user
Parameters
Parameter Type Description Mandatory Default
name string volume name Yes None
capacity int the quota of vol, unit is GB Yes None
owner string the owner of vol, and user ID of a user Yes None
mpCount int the amount of initial meta partitions No 3
enableToken bool whether to enable the token mechanism to control client permissions No false
size int the size of data partitions, unit is GB No 120
followerRead bool enable read from follower No false
crossZone bool cross zone or not. If it is true, parameter zoneName must be empty No false
zoneName string specified zone No default (if crossZone is false)

Delete

curl -v "http://10.196.59.198:17010/vol/delete?name=test&authKey=md5(owner)"

Mark the vol status to MarkDelete first, then delete data partition and meta partition asynchronous, finally delete meta data from persist store.

While deleting the volume, the policy information related to the volume will be deleted from all user information.

Parameters
Parameter Type Description
name string volume name
authKey string calculates the 32-bit MD5 value of the owner field as authentication information

Get

curl -v "http://10.196.59.198:17010/client/vol?name=test&authKey=md5(owner)" | python -m json.tool

Show the base information of the vol, such as name, the detail of data partitions and meta partitions and so on.

Parameters
Parameter Type Description
name string volume name
authKey string calculates the 32-bit MD5 value of the owner field as authentication information

response

{
    "Name": "test",
    "Owner": "user",
    "Status": "0",
    "FollowerRead": "true",
    "MetaPartitions": {},
    "DataPartitions": {},
    "DataPartitions": {},
    "CreateTime": 0
}

Stat

curl -v http://10.196.59.198:17010/client/volStat?name=test

Show the status information of volume.

Parameters
Parameter Type Description
name string volume name

response

{
    "Name": "test",
    "TotalSize": 322122547200000000,
    "UsedSize": 155515112832780000,
    "UsedRatio": "0.48",
    "EnableToken": false
}

Update

curl -v "http://10.196.59.198:17010/vol/update?name=test&capacity=100&authKey=md5(owner)"

Increase the quota of volume, or adjust other parameters.

Parameters
Parameter Type Description Mandatory
name string volume name Yes
authKey string calculates the 32-bit MD5 value of the owner field as authentication information Yes
capacity int the quota of vol, unit is GB Yes
zoneName string update zone name Yes
enableToken bool whether to enable the token mechanism to control client permissions. False by default. No
followerRead bool enable read from follower No

List

curl -v "http://10.196.59.198:17010/vol/list?keywords=test"

List all volumes information, and can be filtered by keywords.

Parameters
Parameter Type Description Mandatory
keywords string get volumes information which contains this keyword No

response

[
   {
       "Name": "test1",
       "Owner": "cfs",
       "CreateTime": 0,
       "Status": 0,
       "TotalSize": 155515112832780000,
       "UsedSize": 155515112832780000
   },
   {
       "Name": "test2",
       "Owner": "cfs",
       "CreateTime": 0,
       "Status": 0,
       "TotalSize": 155515112832780000,
       "UsedSize": 155515112832780000
   }
]

Add Token

curl -v "http://10.196.59.198:17010/token/add?name=test&tokenType=1&authKey=md5(owner)"

Add the token that controls read and write permissions.

Parameters
Parameter Type Description
name string the name of vol
tokenType int 1 is readonly token, 2 is readWrite token
authKey string calculates the 32-bit MD5 value of the owner field as authentication information

Update Token

curl -v "http://10.196.59.198:17010/token/update?name=test&token=xx&tokenType=1&authKey=md5(owner)"

Update token type.

Parameters
Parameter Type Description
name string the name of vol
token string the token value
tokenType int 1 is readonly token, 2 is readWrite token
authKey string calculates the 32-bit MD5 value of the owner field as authentication information

Delete Token

curl -v "http://10.196.59.198:17010/token/delete?name=test&token=xx&authKey=md5(owner)"

Delete specified token.

Parameters
Parameter Type Description
name string the name of vol
token string the token value
authKey string calculates the 32-bit MD5 value of the owner field as authentication information

Get Token

curl -v "http://10.196.59.198:17010/token/get?name=test&token=xx"

Show token information.

Parameters
Parameter Type Description
name string the name of vol
token string the token value

response

{
    "TokenType":2,
    "Value":"siBtuF9hbnNqXzJfMTU48si3nzU4MzE1Njk5MDM1NQ==",
    "VolName":"test"
}

Meta Partition

Create

curl -v "http://10.196.59.198:17010/metaPartition/create?name=test&start=10000"

Split meta partition manually. If max meta partition of the vol which range is [0,end) and end larger than start parameter, old meta partition range will be [0,start], new meta partition will be [start+1,end).

Parameters
Parameter Type Description
name string the name of vol
start uint64 the start value of meta partition which will be create

Get

curl -v "http://10.196.59.198:17010/metaPartition/get?id=1" | python -m json.tool

Show base information of meta partition, such as id, start, end and so on.

Parameters
Parameter Type Description
id uint64 the id of meta partition

response

{
    "PartitionID": 1,
    "Start": 0,
    "End": 9223372036854776000,
    "MaxNodeID": 1,
    "VolName": "test",
    "Replicas": {},
    "ReplicaNum": 3,
    "Status": 2,
    "IsRecover": true,
    "Hosts": {},
    "Peers": {},
    "Zones": {},
    "MissNodes": {},
    "LoadResponse": {}
}

Decommission

curl -v "http://10.196.59.198:17010/metaPartition/decommission?id=13&addr=10.196.59.202:17210"

Remove the replica of meta partition, and create new replica asynchronous.

Parameters
Parameter Type Description
id uint64 the id of meta partition
addr string the addr of replica which will be decommission

Load

curl -v "http://10.196.59.198:17010/metaPartition/load?id=1"

Send load task to the metaNode which meta partition locate on, then check the crc of each replica in the meta partition.

Parameters
Parameter Type Description
id uint64 the id of data partition

Data Partition

Create

curl -v "http://10.196.59.198:17010/dataPartition/create?count=400&name=test"

Create a set of data partition.

Parameters
Parameter Type Description
count int the num of dataPartitions will be create
name string the name of vol

Get

curl -v "http://10.196.59.198:17010/dataPartition/get?id=100"  | python -m json.tool

Get information of the specified data partition.

Parameters
Parameter Type Description
id uint64 the id of data partition

response

{
    "PartitionID": 100,
    "LastLoadedTime": 1544082851,
    "ReplicaNum": 3,
    "Status": 2,
    "Replicas": {},
    "Hosts": {},
    "Peers": {},
    "Zones": {},
    "MissingNodes": {},
    "VolName": "test",
    "VolID": 2,
    "FileInCoreMap": {},
    "FilesWithMissingReplica": {}
}

Decommission

curl -v "http://10.196.59.198:17010/dataPartition/decommission?id=13&addr=10.196.59.201:17310"

Remove the replica of data partition, and create new replica asynchronous.

Parameters
Parameter Type Description
id uint64 the id of data partition
addr string the addr of replica which will be decommission

Load

curl -v "http://10.196.59.198:17010/dataPartition/load?id=1"

Send load task to the dataNode which data partition locate on, then check the crc of each file in the data partition asynchronous.

Parameters
Parameter Type Description
id uint64 the id of data partition

Offline Disk

curl -v "http://10.196.59.198:17010/disk/decommission?addr=10.196.59.201:17310&disk=/cfs1"

Synchronously offline all the data partitions on the disk, and create a new replica for each data partition in the cluster.

Parameters
Parameter Type Description
addr string replica address
disk string disk path

Master Management

Add

curl -v "http://10.196.59.198:17010/raftNode/add?addr=10.196.59.197:17010&id=3"

Add a new master node to master raft group.

Parameters
Parameter Type Description
addr string the addr of master server, format is ip:port
id uint64 the node id of master server

Remove

curl -v "http://10.196.59.198:17010/raftNode/remove?addr=10.196.59.197:17010&id=3"

Remove the master node from master raft group.

Parameters
Parameter Type Description
addr string the addr of master server, format is ip:port
id uint64 the node id of master server

User

Create

curl -H "Content-Type:application/json" -X POST --data '{"id":"testuser","pwd":"12345","type":3}' "http://10.196.59.198:17010/user/create"

Create a user in the cluster to access object storage service. When the cluster starts, the root user is automatically created (the value of type is 0x1).

ChubaoFS treats every Owner of volume as a user. For example, if the value of Owner is testuser when creating a volume, the volume is owned by user testuser.

If there is no user ID with the same value as the Owner when creating the volume, the user named the value of Owner will be automatically created when creating the volume.

body key
Key Type Description Range Mandatory Default
id string user ID Consists of letters, numbers and underscores, no more than 20 characters Yes None
pwd string user’s password Unlimited No ChubaoFSUser
ak string Access Key Consists of 16-bits letters and numbers No Random value
sk string Secret Key Consists of 32-bits letters and numbers No Random value
type int user type 2: [admin] / 3: [normal user] Yes None

Delete

curl -v "http://10.196.59.198:17010/user/delete?user=testuser"

Delete the specified user in the cluster.

Parameters
Parameter Type Description
user string user ID

Get

Show basic user information, including user ID, Access Key, Secret Key, list of owned volumes, list of permissions granted by other users, user type, creation time.

The field policy shows the volumes which the user has permission, of which own_vols indicates that volumes owned by the user, and authorized_vols indicates the volume authorized by other users to the user with restrictions.

There are two ways to obtain:

Query by User ID
curl -v "http://10.196.59.198:17010/user/info?user=testuser" | python -m json.tool
Parameters
Parameter Type Description
user string user ID
Query by Access Key
curl -v "http://10.196.59.198:17010/user/akInfo?ak=0123456789123456" | python -m json.tool
Parameters
Parameter Type Description
ak string Access Key

response

{
     "user_id": "testuser",
     "access_key": "0123456789123456",
     "secret_key": "ZVY5RHlrnOrCjImW9S3MajtYZyxSegcf",
     "policy": {
         "own_vols": ["vol1"],
         "authorized_vols": {
             "ltptest": [
                 "perm:builtin:ReadOnly",
                 "perm:custom:PutObjectAction"
             ]
         }
     },
     "user_type": 3,
     "create_time": "2020-05-11 09:25:04"
}

List Users

curl -v "http://10.196.59.198:17010/user/list?keywords=test" | python -m json.tool

Query information about all users in a cluster whose user ID contains the keyword.

Parameters
Parameter Type Description
keywords string check user ID contains this or not

Update

curl -H "Content-Type:application/json" -X POST --data '{"user_id":"testuser","access_key":"KzuIVYCFqvu0b3Rd","secret_key":"iaawlCchJeeuGSnmFW72J2oDqLlSqvA5","type":3}' "http://10.196.59.198:17010/user/update"

Update the specified user’s information, including access key, secret key and user type.

body key
Key Type Description Mandatory
user_id string user ID value after updating Yes
access_key string Access Key value after updating No
secret_key string Secret Key value after updating No
type int user type value after updating No

Update Permission

curl -H "Content-Type:application/json" -X POST --data '{"user_id":"testuser","volume":"vol","policy":["perm:builtin:ReadOnly","perm:custom:PutObjectAction"]}' "http://10.196.59.198:17010/user/updatePolicy"

Update the specified user’s permission to a volume. There are three types of values for policy:

  • Grant read-only or read-write permission, the value is perm:builtin:ReadOnly or perm:builtin:Writable.
  • Grant a permission of the specified action, the format is action:oss:XXX, take GetObject action as an example, the value of policy is action:oss:GetObject.
  • Grant a custom permission, the format is perm:custom:XXX, where XXX is customized by the user.

After the permissions are specified, the user can only access the volume within the specified permissions when using the object storage. If the user already has permissions for this volume, this operation will overwrite the original permissions.

body key
Key Type Description Mandatory
user_id string user ID to be set Yes
volume string volume name to be set Yes
policy string slice policy to be set Yes

Remove Permission

curl -H "Content-Type:application/json" -X POST --data '{"user_id":"testuser","volume":"vol"}' "http://10.196.59.198:17010/user/removePolicy"

Remove all permissions of a specified user for a volume.

body key
Key Type Description Mandatory
user_id string user ID to be deleted Yes
volume string volume name to be deleted Yes

Transfer Volume

curl -H "Content-Type:application/json" -X POST --data '{"volume":"vol","user_src":"user1","user_dst":"user2","force":"true"}' "http://10.196.59.198:17010/user/transferVol"

Transfer the ownership of the specified volume. This operation removes the specified volume from the owner_vols of source user name and adds it to the owner_vols of target user name; At the same time, the value of the field Owner in the volume structure will also be updated to the target user ID.

body key
Key Type Description Mandatory
volume string Volume name to be transfered Yes
user_src string Original owner of the volume, and must be the same as the Owner of the volume Yes
user_dst string Target user ID after transferring Yes
force bool Force to transfer the volume. If the value is set to true, even if the value of user_src is different from the value of the owner of the volume, the volume will also be transferred to the target user No

Meta Node API

Meta Partition

Get Partitions

curl -v  http://10.196.59.202:17210/getPartitions

Get all meta-partition base information of the metanode.

Get Partition by ID

curl -v http://10.196.59.202:17210/getPartitionById?pid=100

Get the specified partition information, this result contains: leader address, raft group peer and cursor.

Parameters
Parameter Type Description
pid integer meta-partition id

Inode

Get Inode

curl -v http://10.196.59.202:17210/getInode?pid=100&ino=1024

Get inode information

Parameters
Parameter Type Description
pid integer meta-partition id
ino integer inode id

Get Extents by Inode

curl -v http://10.196.59.202:17210/getExtentsByInode?pid=100&ino=1024

Get inode all extents information

Parameters
Parameter Type Description
pid integer meta-partition id
ino integer inode id

Get All Inodes

curl -v http://10.196.59.202:17210/getAllInodes?pid=100

Get all inodes of the specified partition

Parameters
Parameter Type Description
pid integer meta-partition id

Dentry

Get Dentry

curl -v 'http://10.196.59.202:17210/getDentry?pid=100&name=""&parentIno=1024'

Get dentry information

Parameters
Parameter Type Description
pid integer meta partition id
name string file or directory name
parentIno integer file or directory parent directory inode

Get Directory

curl -v "http://10.196.59.202:17210/getDirectory?pid=100&parentIno=1024"

Get all files of the parent inode is 1024

Parameters
Parameter Type Description
pid integer partition id
ino integer inode id

Get All Dentry

curl -v "http://10.196.59.202:17210/getAllDentry?pid=100"
Parameters
Parameter Type Description
pid integer partition id

Command Line Interface

CLI

Using the Command Line Interface (CLI) can manage the cluster conveniently. With this tool, you can view the status of the cluster and each node, and manage each node, volumes, or users.

As the CLI continues to improve, it will eventually achieve 100% coverage of the APIs of each node in the cluster.

Compile and Configure

After downloading the ChubaoFS source code, execute the command go build in the directory chubaofs/cli to generate cli.

At the same time, a configuration file named .cfs-cli.json will be generated in the directory root, and the master address can be changed to the current cluster master address. You can also get or set the master address by executing the command ./cli config info or ./cli config set.

Bug Shooting

The logs of cfs-cli tool are in the directory /tmp/cfs/cli, which offer detail running information for bug shooting.

Usage

In the directory chubaofs/cli, execute the command ./cli --help or ./cli -h to get the CLI help document.

CLI is mainly divided into seven types of management commands.

Commands List
Command description
cli cluster Manage cluster components
cli metanode Manage meta nodes
cli datanode Manage data nodes
cli datapartition Manage data partitions
cli metapartition Manage meta partitions
cli config Manage configuration for cli tool
cli completion Generating bash completions
cli volume, vol Manage cluster volumes
cli user Manage cluster users
cli compatibility Compatibility test
Cluster Management
./cli cluster info          #Show cluster summary information

./cli cluster stat          #Show cluster status information

./cli cluster freeze [true/false]        #Turn on or turn off the automatic allocation of the data partitions.

./cli cluster threshold [float]     #Set the threshold of memory on each meta node.
MetaNode Management
./cli metanode list          #List information of meta nodes

./cli metanode info [Address]     #Show detail information of a meta node

./cli metanode decommission [Address] #Decommission partitions in a meta node to other nodes
DataNode Management
./cli datanode list          #List information of data nodes

./cli datanode info [Address]         #Show detail information of a data node

./cli datanode decommission [Address]   #Decommission partitions in a data node to other nodes
DataPartition Management
./cli datapartition info [Partition ID]        #Display detail information of a data partition

./cli datapartition decommission [Address] [Partition ID]   #Decommission a replication of the data partition to a new address

./cli datapartition add-replica [Address] [Partition ID]    #Add a replication of the data partition on a new address

./cli datapartition del-replica [Address] [Partition ID]    #Delete a replication of the data partition from a fixed address

./cli datapartition check    #Diagnose partitions, display the partitions those are corrupt or lack of replicas
MetaPartition Management
./cli metapartition info [Partition ID]        #Display detail information of a meta partition

./cli metapartition decommission [Address] [Partition ID]   #Decommission a replication of the meta partition to a new address

./cli metapartition add-replica [Address] [Partition ID]    #Add a replication of the meta partition on a new address

./cli metapartition del-replica [Address] [Partition ID]    #Delete a replication of the meta partition from a fixed address

./cli metapartition check    #Diagnose partitions, display the partitions those are corrupt or lack of replicas
Config Management
./cli config info     #Show configurations of cli

./cli config set [flags]    #Set configurations of cli
Flags:
    --addr      string      #Specify master address [{HOST}:{PORT}]
    --timeout   uint16      #Specify timeout for requests [Unit: s] (default 60)
Completion Management
./cli completion      #Generate bash completions
Volume Management
./cli volume create [VOLUME NAME] [USER ID] [flags]     #Create a new volume
Flags:
    --capacity uint                                     #Specify volume capacity [Unit: GB] (default 10)
    --dp-size  uint                                     #Specify size of data partition size [Unit: GB] (default 120)
    --follower-read                                     #Enable read form replica follower (default true)
    --mp-count int                                      #Specify init meta partition count (default 3)
    -y, --yes                                           #Answer yes for all questions

./cli volume delete [VOLUME NAME] [flags]               #Delete a volume from cluster
Flags:
    -y, --yes                                           #Answer yes for all questions

./cli volume info [VOLUME NAME] [flags]                 #Show volume information
Flags:
    -d, --data-partition                                #Display data partition detail information
    -m, --meta-partition                                #Display meta partition detail information

./cli volume add-dp [VOLUME] [NUMBER]                   #Create and add more data partition to a volume

./cli volume list                                       #List cluster volumes

./cli volume transfer [VOLUME NAME] [USER ID] [flags]   #Transfer volume to another user. (Change owner of volume)
Flags：
    -f, --force                                         #Force transfer without current owner check
    -y, --yes                                           #Answer yes for all questions
User Management
./cli user create [USER ID] [flags]         #Create a new user
Flags：
    --access-key string                     #Specify user access key for object storage interface authentication
    --secret-key string                     #Specify user secret key for object storage interface authentication
    --password string                       #Specify user password
    --user-type string                      #Specify user type [normal | admin] (default "normal")
    -y, --yes                               #Answer yes for all questions

./cli user delete [USER ID] [flags]         #Delete specified user
Flags：
    -y, --yes                               #Answer yes for all questions

./cli user info [USER ID]                   #Show detail information about specified user

./cli user list                             #List cluster users

./cli user perm [USER ID] [VOLUME] [PERM]   #Setup volume permission for a user
                                            #The value of [PERM] is READONLY, RO, READWRITE, RW or NONE

./cli user update [USER ID] [flags]         #Update information about specified user
Flags：
    --access-key string                     #Update user access key
    --secret-key string                     #Update user secret key
    --user-type string                      #Update user type [normal | admin]
    -y, --yes                               #Answer yes for all questions
Compatibility Test
./cli cptest meta [Snapshot Path] [Host] [Partition ID]         #Metadata compatibility test
Parameters：
        [Snapshot Path] string                     #The path which snapshot file located
        [Host] string                              #The metanode host which generated the snapshot file
        [Partition ID] string                      #The meta partition ID which to be compared

Example:

  1. Use the old version to prepare metadata, stop writing metadata,after waiting for the latest snapshot to be generated(about 5 minutes), copy the snapshot file to the local machine
  2. Execute the metadata comparison command on local machine
[Verify result]
All dentry are consistent
All inodes are consistent
All meta has checked

Use Cases

ChubaoFS is a distributed file system that is compatible with most POSIX file system semantics. When ChubaoFS is mounted, it can be as simple as using a local file system. Basically, it can be used in any case where a file system is needed, replacing the local file system, and realizing infinitely expandable storage without physical boundaries. It has been applied in various scenarios, and the following are some of the extracted scenes.

Machine Learning

Disadvantages of using a local disk to store training data sets:

  • The local disk space is small, and there are multiple models. The training data set of each model reaches the TB level. If you use a local disk to store the training data set, you need to reduce the size of the training data set.
  • Training data sets need to be updated frequently, requiring more disk space.
  • Risk of loss of training data set if machine crashes.

The advantages of using ChubaoFS to store training data sets:

  • Unlimited disk space, easy to expand capacity. It can automatically expand disk capacity according to the percentage of disk usage, enabling storage system capacity expansion on demand, greatly saving storage costs.
  • Multiple replicas of data to ensure high data reliability without worrying about losing data.
  • Compatible with POSIX file system interface, no changes required by the application.

ElasticSearch

Using local disks to store data often encounters the following problems:

  • Disk usage is uneven and disk IO cannot be fully utilized.
  • Local disk space is limited.

The advantages of using ChubaoFS as a backend storage:

  • Unlimited disk space, easy to expand capacity. It can automatically expand disk capacity according to the percentage of disk usage, enabling storage system capacity expansion on demand, greatly saving storage costs.
  • Disk IO usage is uniform and disk IO is fully utilized.
  • Ensure data is highly reliable without worrying about losing data.

Nginx Log Storage

Do you often worry about running out of local disk space? With ChubaoFS, you can store datas in a distributed file system without worrying about running out of disk space. If you use a local disk to store logs, you may often worry about the following issues:

  • Docker local disk space is small.
  • If the docker container crashes, the log is lost and unrecoverable.
  • The mixed deployment of physical machine and docker machine is difficult to manage and has high operation and maintenance cost.

The advantages of using ChubaoFS to store nginx logs are:

  • The disk space is unlimited and easy to expand. The disk capacity is automatically expanded according to the percentage of disk usage. The storage system can be expanded on demand, which greatly saves storage costs.
  • Ensure data is highly reliable and do not worry about losing data.
  • Multiple replicas to solve the problem of unable to write to the log caused by disk error and datanode crashes.
  • Compatible with the POSIX file system interface, no changes required by the application.
  • The operation and maintenance of ChubaoFS are simple, and one person can easily manage the cluster of tens of thousands of machines.

Spark

In the big data set scenario, are you worried about the amount of data stored in Spark intermediate calculation results in order to carefully calculate each task? You can store the shuffle results to ChubaoFS, and no longer worry about the disk has no free space which causes the task to fail. This enables the separation of storage and computation. The pain points of using local disk to store shuffle intermediate results:

  • Insufficient disk space.
  • Too many temporary directory files to create new files.

The advantages of using ChubaoFS:

  • Unlimited disk space, easy to expand capacity. It can automatically expand disk capacity according to the percentage of disk usage, enabling storage system capacity expansion on demand, greatly saving storage costs.
  • Meta node manages file metadata, which can be expanded horizontally and the number of files is unlimited.

MySQL Database Backup

Disadvantages of using OSS(Object Storage Service) to back up MySQL database:

  • Need to use the OSS SDK or RESTful API to develop backup programs which increases the difficulty of operation and maintenance.
  • If backup file fails, troubleshooting is more difficult.
  • After backing up files to OSS, it is not convenient to check whether the files are successfully uploaded.
  • Backup files are processed by multiple layers of services, which affects performance.

Advantages of using ChubaoFS to back up MySQL databases:

  • Easy to use, compatible with POSIX file interface, and can be used as a local file system.
  • Complete and detailed operation logs are stored in the local file system, making it easy to troubleshoot problems.
  • Simply execute the ls command to verify that the file was successfully uploaded.
  • Supports PageCache and WriteCache, and file read and write performance is significantly improved compared to OSS.

Performance

Environment

Cluster Information

Instance Nodes CPU Memory Storage Network Description
Master 3 32 32 GB 120 GB SSD 10 Gb/s  
MetaNode 10 32 32 GB 16 x 1TB SSD 10 Gb/s hybrid deployment
DataNode 10 32 32 GB 16 x 1TB SSD 10 Gb/s hybrid deployment

Volume Setup

Parameter Default Recommend Description
FollowerRead True True  
Capacity 10 GB 300 000 000 GB  
Data Replica Number 3 3  
Meta Replica Number 3 3  
Data Partition Size 120 GB 120 GB Logical upper limit with no pre-occupied space.
Data Partition Count 10 1500  
Meta Partition Count 3 10  
Cross Zone False False  

Set volume parameters by following:

$ cfs-cli volume create test-vol {owner} --capacity=300000000 --mp-count=10
Create a new volume:
  Name                : test-vol
  Owner               : ltptest
  Dara partition size : 120 GB
  Meta partition count: 10
  Capacity            : 300000000 GB
  Replicas            : 3
  Allow follower read : Enabled

Confirm (yes/no)[yes]: yes
Create volume success.

$ cfs-cli volume add-dp test-vol 1490

client configuration

Parameter Default Recommend
rate limit -1 -1 
#get current iops, default:-1(no limits on iops)：
$ http://[ClientIP]:[ProfPort]/rate/get
#set iops
$ http://[ClientIP]:[ProfPort]/rate/set?write=800&read=800

Small File Performance and Scalability

Small file operation performance and scalability benchmark test by mdtest.

Setup

#!/bin/bash
set -e
TARGET_PATH="/mnt/test/mdtest" # mount point of ChubaoFS volume
for FILE_SIZE in 1024 2048 4096 8192 16384 32768 65536 131072 # file size
do
mpirun --allow-run-as-root -np 512 --hostfile hfile64 mdtest -n 1000 -w $i -e $FILE_SIZE -y -u -i 3 -N 1 -F -R -d $TARGET_PATH;
done

Benchmark

Small File Benchmark
File Size (KB) 1 2 4 8 16 32 64 128
Creation (TPS) 70383 70383 73738 74617 69479 67435 47540 27147
Read (TPS) 108600 118193 118346 122975 116374 110795 90462 62082
Removal (TPS) 87648 84651 83532 79279 85498 86523 80946 84441
Stat (TPS) 231961 263270 264207 252309 240244 244906 273576 242930

IO Performance and Scalability

IO Performance and benchmark scalability test by fio.

Note: Multiple clients mount the same volume. And the process refers to the fio process.

1. Sequential Read

Setup

#!/bin/bash
fio -directory={} \
    -ioengine=psync \
    -rw=read \  # sequential read
    -bs=128k \  # block size
    -direct=1 \ # enable direct IO
    -group_reporting=1 \
    -fallocate=none \
    -time_based=1 \
    -runtime=120 \
    -name=test_file_c{} \
    -numjobs={} \
    -nrfiles=1 \
    -size=10G

Bandwidth(MB/s)

Sequential Read Bandwidth (MB/s)
  1 Process 4 Processes 16 Processes 64 Processes
1 Client 148.000 626.000 1129.000 1130.000
2 Clients 284.000 1241.000 2258.000 2260.000
4 Clients 619.000 2640.000 4517.000 4515.000
8 Clients 1193.000 4994.000 9006.000 9034.000

IOPS

Sequential Read IOPS
  1 Process 4 Processes 16 Processes 64 Processes
1 Client 1180.000 5007.000 9031.000 9040.000
2 Clients 2275.000 9924.000 18062.000 18081.000
4 Clients 4954.000 21117.000 36129.000 36112.000
8 Clients 9531.000 39954.000 72048.000 72264.000

Latency(Microsecond)

Sequential Read Latency (Microsecond)
  1 Process 4 Processes 16 Processes 64 Processes
1 Client 842.200 794.340 1767.310 7074.550
2 Clients 874.255 801.690 1767.370 7071.715
4 Clients 812.363 760.702 1767.710 7077.065
8 Clients 837.707 799.851 1772.620 7076.967

2. Sequential Write

Setup

#!/bin/bash
fio -directory={} \
    -ioengine=psync \
    -rw=write \ # sequential write
    -bs=128k \  # block size
    -direct=1 \ # enable direct IO
    -group_reporting=1 \
    -fallocate=none \
    -name=test_file_c{} \
    -numjobs={} \
    -nrfiles=1 \
    -size=10G

Bandwidth(MB/s)

Sequential Write Bandwidth (MB/s)
  1 Process 4 Processes 16 Processes 64 Processes
1 Client 52.200 226.000 956.000 1126.000
2 Clients 104.500 473.000 1763.000 2252.000
4 Clients 225.300 1015.000 2652.000 3472.000
8 Clients 480.600 1753.000 3235.000 3608.000

IOPS

Sequential Write IOPS
  1 Process 4 Processes 16 Processes 64 Processes
1 Client 417 1805 7651 9004
2 Clients 835 3779 14103 18014
4 Clients 1801 8127 21216 27777
8 Clients 3841 14016 25890 28860

Latency(Microsecond)

Sequential Write Latency (Microsecond)
  1 Process 4 Processes 16 Processes 64 Processes
1 Client 2385.400 2190.210 2052.360 7081.320
2 Clients 2383.610 2081.850 2233.790 7079.450
4 Clients 2216.305 1947.688 2946.017 8842.903
8 Clients 2073.921 2256.120 4787.496 17002.425

3. Random Read

Setup

#!/bin/bash
fio -directory={} \
    -ioengine=psync \
    -rw=randread \ # random read
    -bs=4k \       # block size
    -direct=1 \    # enable direct IO
    -group_reporting=1 \
    -fallocate=none \
    -time_based=1 \
    -runtime=120 \
    -name=test_file_c{} \
    -numjobs={} \
    -nrfiles=1 \
    -size=10G

Bandwidth(MB/s)

Random Read Bandwidth (MB/s)
  1 Process 4 Processes 16 Processes 64 Processes
1 Client 6.412 39.100 216.000 534.000
2 Clients 14.525 88.100 409.000 1002.000
4 Clients 33.242 200.200 705.000 1693.000
8 Clients 59.480 328.300 940.000 2369.000

IOPS

Random Read IOPS
  1 Process 4 Processes 16 Processes 64 Processes
1 Client 1641 10240 56524.800 140288
2 Clients 3718 23142.4 107212.8 263168
4 Clients 8508 52428.8 184627.2 443392
8 Clients 15222 85072.8 246681.6 621056

Latency(Microsecond)

Random Read Latency (Microsecond)
  1 Process 4 Processes 16 Processes 64 Processes
1 Client 603.580 395.420 287.510 466.320
2 Clients 532.840 351.815 303.460 497.100
4 Clients 469.025 317.140 355.105 588.847
8 Clients 524.709 382.862 530.811 841.985

4. Random Write

Setup

#!/bin/bash
fio -directory={} \
    -ioengine=psync \
    -rw=randwrite \ # random write
    -bs=4k \        # block size
    -direct=1 \     # enable direct IO
    -group_reporting=1 \
    -fallocate=none \
    -time_based=1 \
    -runtime=120 \
    -name=test_file_c{} \
    -numjobs={} \
    -nrfiles=1 \
    -size=10G

Bandwidth(MB/s)

Random Write Bandwidth (MB/s)
  1 Process 4 Processes 16 Processes 64 Processes
1 Client 3.620 17.500 118.000 318.000
2 Clients 7.540 44.800 230.000 476.000
4 Clients 16.245 107.700 397.900 636.000
8 Clients 39.274 208.100 487.100 787.100

IOPS

Random Write IOPS
  1 Process 4 Processes 16 Processes 64 Processes
1 Client 926.000 4476.000 31027.200 83251.200
2 Clients 1929.000 11473.000 60313.600 124620.800
4 Clients 4156.000 27800.000 104243.200 167014.400
8 Clients 10050.000 53250.000 127692.800 206745.600

Latency

Random Write Latency
  1 Process 4 Processes 16 Processes 64 Processes
1 Client 1073.150 887.570 523.820 784.030
2 Clients 1030.010 691.530 539.525 1042.685
4 Clients 955.972 575.183 618.445 1552.205
8 Clients 789.883 598.393 1016.185 2506.424

Metadata Performance and Scalability

Metadata performance and scalability benchmark test by mdtest.

Setup

#!/bin/bash
TEST_PATH=/mnt/cfs/mdtest # mount point of ChubaoFS volume
for CLIENTS in 1 2 4 8 # number of clients
do
mpirun --allow-run-as-root -np $CLIENTS --hostfile hfile01 mdtest -n 5000 -u -z 2 -i 3 -d $TEST_PATH;
done

Dir Creation

Dir Creation
  1 Process 4 Processes 16 Processes 64 Processes
1 Client 448.618 2421.001 14597.97 43055.15
2 Clients 956.947 5917.576 28930.431 72388.765
4 Clients 2027.02 13213.403 54449.056 104771.356
8 Clients 4643.755 27416.904 89641.301 119542.62

Dir Removal

Dir Removal
  1 Process 4 Processes 16 Processes 64 Processes
1 Client 399.779 2118.005 12351.635 34903.672
2 Clients 833.353 5176.812 24471.674 50242.973
4 Clients 1853.617 11462.927 46413.313 91128.059
8 Clients 4441.435 24133.617 74401.336 115013.557

Dir Stat

Dir Stat
  1 Process 4 Processes 16 Processes 64 Processes
1 Client 283232.761 1215309.524 4231088.104 12579177.02
2 Clients 572834.143 2169669.058 8362749.217 18120970.71
4 Clients 1263474.549 3333746.786 10160929.29 31874265.88
8 Clients 2258670.069 8715752.83 22524794.98 77533648.04

File Creation

File Creation
  1 Process 4 Processes 16 Processes 64 Processes
1 Client 448.888 2400.803 13638.072 27785.947
2 Clients 925.68 5664.166 25889.163 50434.484
4 Clients 2001.137 12986.968 50330.952 91387.825
8 Clients 4479.831 25933.437 86667.966 112746.199

File Removal

File Removal
  1 Process 4 Processes 16 Processes 64 Processes
1 Client 605.143 3678.138 18631.342 47035.912
2 Clients 1301.151 8365.667 34005.25 64860.041
4 Clients 3032.683 14017.426 50938.926 80692.761
8 Clients 7170.386 32056.959 68761.908 88357.563

Tree Creation

Tree Creation
  1 Process 4 Processes 16 Processes 64 Processes
1 Client 305.778 229.562 86.299 23.917
2 Clients 161.31 211.119 76.301 24.439
4 Clients 260.562 223.153 81.209 23.867
8 Clients 350.038 220.744 81.621 17.144

Tree Removal

Tree Removal
  1 Process 4 Processes 16 Processes 64 Processes
1 Client 137.462 70.881 31.235 7.057
2 Clients 217.026 113.36 23.971 7.128
4 Clients 231.622 113.539 30.626 7.299
8 Clients 185.156 83.923 20.607 5.515

Integrity

  • Linux Test Project / fs

Workload

  • Database backup

  • Java application logs

  • Code git repo

  • Database systems

    MyRocks, MySQL Innodb, HBase,

Scalability

  • Volume Scalability: tens to millions of cfs volumes
  • Metadata Scalability: a big volume with billions of files/directories

Contributing to ChubaoFS

Bug Reports

Please make sure the bug is not already reported by searching the repository with reasonable keywords. Then, open an issue with steps to reproduce.

Patch Guidelines

In order for the patch to be accepted with higher possibility, there are a few things you might want to pay attention to:

runltp -f fs -d [MOUNTPOINT]
  • A good commit message describing the bug fix or the new feature is preferred.
  • DCO is required, so please add Signed-off-by to the commit.

Credits

Sections of this documents have been borrowed from CoreDNS and Fluentd

Environment Requirements and Capacity Planning

Environment Requirements

The following table lists the system and hardware requirements of the performance test environment and production environment. You can also refer to the capacity planning chapter to accurately customize the deployment plan based on your cluster’s actual capacity planning. Note that since the DataNode used some features of linux kernal, so that the kernel version of servers which used for deploy DataNode must be later than 3.10.

In order to speed up read and write of meta data, the meta data is stored in memory, while the DataNode mainly occupies disk resources. To maximize the use of node resources, you can mix-deploy DataNode and MetaNode on the same node.

Role Spec Test Product
Master CPU >=4C >=8C
  Memory >=4G >=16G
  Kernel >=3.10 >=3.10
  Nodes 3 3
DataNode CPU >=4C >=4C
  Memory >=4G >=8G
  Kernel >=3.10 >=3.10
  Disk Capacity >=1TB >=2TB
  Disk Type sata | ssd sata | ssd
  File System xfs | etx4 xfs | etx4
  Nodes >=3 100~1000
MetaNode CPU >=4C >=8C
  Memory >=8G >=16G
  Kernel >=3.10 >=3.10
  Nodes >=4 100~1000
Client CPU >=2C >=2C
  Memory >=4G >=1G
  Kernel >=3.10 >=3.10

Capacity Planning

First of all, you have to assess the highest expected number of files and storage capacity of the cluster in the future. Secondly, you need to know the machine resources you currently have, and the total memory, CPU cores, and disks on each machine. If you have been clear about those statistics, you can use the empirical reference values ​​given in the second section to see which scale your current environment belongs to, what file size it can carry,or you need to prepare for the current file experience requirements How many resources to prevent frequent expansion of machine resources.

Total File Count Total File Size Total memory Total Disk Space
1,000,000,000 10PB 2048 GB 10PB

The higher the proportion of large files, the greater the MetaNode pressure.

Of course, if you feel that the current resources are adequately used, you don’t need to meet the capacity growth requirements all at once. Then you can pay attention to the capacity warning information of MetaNode/DataNode in time. When the memory or disk is about to run out, dynamically increase MetaNode/DataNode to adjust the capacity. In other words, if you find that the disk space is not enough, you can increase the disk or increase DataNode. If you find that all MetaNode memory is too full, you can increase MetaNode to relieve memory pressure.

Multi-Zone Deploy

If you want the cluster to support fault tolerance in the computer room, you can deploy a ChubaoFS cluster across computer rooms. At the same time, it should be noted that since the communication delay between computer rooms is higher than that of a single computer room, if the requirements for high availability are greater than low latency, you can choose a cross-computer room deployment solution. If you have higher performance requirements, it is recommended to deploy clusters in a single computer room. Configuration scheme: Modify the zoneName parameter in the DataNode/MetaNode configuration file, specify the name of the computer room where you are, and then start the DataNode/MetaNode process, the computer room will be stored and recorded by the Master along with the registration of DataNode/MetaNode.

Create a single zone volume:

$ cfs-cli volume create {name} --zone-name={zone}

In order to prevent volume initialization failure in a single computer room, please ensure that the DataNode of a single computer room is not less than 3 and MetaNode is not less than 4.

Create a cross-zone volume:

$ cfs-cli volume create {name} --cross-zone=true

Q&A

  • If you are new to ChubaoFS and want to start quickly, please refer to Run Cluster on Docker
  • If you are interested in ChubaoFS and want to conduct a performance test before applying it in the product environment, please refer to Performance
  • If you have completed the assessment of ChubaoFS and want to put it into product environment, and want to learn how to carry out capacity planning and environment preparing, please refer to Environment Requirements and Capacity Planning
  • If you want to know about best practices of ChubaoFS, please refer to Use Cases
  • If you encounter some problems in the product environment, the following content may help to solve your problems.

For the convenience of description, we define the following keyword abbreviations

Full Name Abbreviation
Data Partition dp
Meta Partition mp
Data Partition Replica dpr
Meta Partition Replica mpr
NodeSet ns
DataNode dn
MetaNode mn

Compile

  1. Compile ChubaoFS on one machine, but it cannot be started when deployed to other machines.

First please make sure to use the PORTABLE=1 make static_lib command to compile RocksDB, then use the ldd command to check whether the dependent libraries are installed on the machine. After installing the missing libraries, execute the ldconfig command.

  1. A problem similar to undefined reference to ‘ZSTD_versionNumber’.

There are two solutions.

  • It can be compiled by adding the specified library to CGO_LDFLAGS.

For example: CGO_LDFLAGS="-L/usr/local/lib -lrocksdb -lzstd". This requires that the zstd library is also installed on other deployment machines.

  • Remove the script that automatically detects whether the ‘zstd’ library is installed.

Example of file location: rockdb-5.9.2/build_tools/build_detect_platform

The deleted content is as follows.

# Test whether zstd library is installed
    $CXX $CFLAGS $COMMON_FLAGS -x c++ - -o /dev/null 2>/dev/null  <<EOF
    #include <zstd.h>
    int main() {}
EOF
    if [ "$?" = 0 ]; then
        COMMON_FLAGS="$COMMON_FLAGS -DZSTD"
        PLATFORM_LDFLAGS="$PLATFORM_LDFLAGS -lzstd"
        JAVA_LDFLAGS="$JAVA_LDFLAGS -lzstd"
    fi

Node & Disk Failure

If a node or disk fails, you can take the failed node or disk offline by decommission command.

  1. Decommission Datanode/Metanode

    $ cfs-cli metanode/datanode decommission 192.168.0.21:17210
    • At what time to decommission mn/dn？

    If a node fails and cannot reboot, it has to be removed from the cluster and the partitions on this machine is automatically migrated to other healthy nodes.

    • The decommission caused an acute increasing on disk io and network io.

    The decommission will trigger automatic partition migration witch consumes a lot of network resources. Therefore, try to execute the decommission during off peak hours and avoid taking multiple nodes decommission at the same time.

    • What is the sign of decommission completion？

      $ cfs-cli datapartition check

      If bad partition ids is empty, the decommission is done.

      • Common error1: There is no mn available, all mn memory or disks are full at this time, new mn needs to be added to the cluster
      • Common error2: The port number is wrong, the identifier of each mn should be a combination of ip+port , and port is the listen port in the mn configuration file, which cannot be wrong

  2. Decommission Disk

If a disk fails but the node is healthy, you can partially decommission the disk. Similarly, as with dn/mn decommission, please try to avoid multiple decommission operations during off peak hours.

$ cfs-cli disk decommission {disk} 192.168.0.11:17310

If correct, all the dp on the disk is migrated to other disks or nodes.

The common error is similar to decommission dn.

  1. Decommission Data Partition/Meta Partition

    $ cfs-cli datapartition decommission 192.168.0.11:17310 {Partition ID}

If correct, the dp is migrated to other nodes.

  • At what time to decommission a partition？

    • There is too many partitions on one node, decommission some partitions to reduce the pressure on this node.
    • Take a small step instead of dn/mn decommission to prevent overloading of the cluster.

  • The common error is similar to decommission dn/mn.

  1. If the disk is full, will it explode?

It is recommended set reserved space in the dn startup json file, it is behind the disk path parameter, disk: “{disk path}:{reserved space}”. If the remaining space is less than reserved space, the dn turns readonly.

Performance of Data and Metadata

  1. How does ChubaoFS compare with its alternatives ?
  • Ceph

Ceph (https://ceph.com/) is a widely-used free-software storage platform. It can be configured for object storage, block storage, as well as file system. But many nice features provided by Ceph (e.g., various storage backends) also make it very complicated and difficult to learn and deploy. In addition, in certain scenarios such as metadata operations and small file operations, its performance in a multi-client environment can be hard to optimize.

  • GFS and HDFS

GFS and its open source implementation HDFS (https://github.com/apache/hadoop) are designed for storing large files with sequential access. Both of them adopt the master-slave architecture, where the single master stores all the file metadata. Unlike GFS and HDFS, ChubaoFS employs a separate metadata subsystem to provide a scalable solution for the metadata storage so that the resource manager has less chance to become the bottleneck.

  • Hadoop Ozone

Hadoop Ozone is a scalable distributed object storage system designed for Hadoop. It was originally proposed to solve the problem of HDFS namenode expansion. It reconstructs the namenode metadata management part of hdfs and reuses the datanode of hdfs. ChubaoFS has many of the same design concepts like ozone such as: supporting for volume isolation, compatible with both raft/master-slave synchronous replication mechanisms, implenting for s3-compatible interfaces. In addition, ChubaoFS’s POSIX fuse-client interface supports random file reading and writing, and optimizes reading and writing of small files.

  • Haystack

Haystack from Facebook takes after log-structured filesystems to serve long tail of requests seen by sharing photos in a large social network. The key insight is to avoid disk operations when accessing metadata. ChubaoFS adopts similar ideas by putting the file metadata into the main memory.

However, different from Haystack, the actually physical offsets instead of logical indices of the file contents are stored in the memory, and deleting a file is achieved by the punch hole interface provided by the underlying file system instead of relying on the garbage collector to perform merging and compacting regularly for more efficient disk utilization. In addition, Haystack does not guarantee the strong consistency among the replicas when deleting the files, and it needs to perform merging and compacting regularly for more efficient disk utilization, which could be a performance killer. ChubaoFS takes a different design principle to separate the storage of file metadata and contents. In this way, we can have more flexible and cost-effective deployments of meta and data nodes.

  • Public Cloud

Windows Azure Storage (https://azure.microsoft.com/en-us/) is a cloud storage system that provides strong consistency and multi-tenancy to the clients. Different from ChubaoFS, it builds an extra partition layer to handle random writes before streaming data into the lower level. AWS EFS (https://aws.amazon.com/efs/) is a cloud storage service that provides scalable and elastic file storage. Depending on the use cases, there could be a considerable amount of cost associated with using these cloud storage services.

  • Others

GlusterFS (https://www.gluster.org/) is a scalable distributed file system that aggregates disk storage resources from multiple servers into a single global namespace. MooseFS (https://moosefs.com/) is a fault- tolerant, highly available, POSIX-compliant, and scalable distributed file system. However, similar to HDFS, it employs a single master to manage the file metadata.

  1. If the scale of metadata is huge, how to improve cluster performance?

The metadata of ChubaoFS is stored in the memory. Expanding memory of the mn or expanding the amount of mn horizontally will significantly improve the metadata performance and support a large number of small files.

  1. If a dn/mn is added to the cluster, will it be automatically rebalanced, for example the dp/mp on the old node are migrated to the new node?

No. Considering that the rebalance may cause overloading and risks data loss, it will not automatically rebalance. If you want the new node to carry more dp/mp to disperse the pressure of the old node, you can create new dp for this volume and the new dp may locate on new nodes, or you can decommission the dp on the old node.

  1. There are scenes of batch delete files witch cause cluster overloads

You can set and view the background file deletion rate by following command, the default value is 0, which means unlimited. It is recommended to set markdeleterate=1000, and then dynamically adjust it according to the cpu status of the nodes in the cluster.

$ cfs-cli cluster info
$ cfs-cli cluster delelerate -h
 Set delete parameters

 Usage:
   cfs-cli cluster delelerate [flags]

 Flags:
       --auto-repair-rate string         DataNode auto repair rate
       --delete-batch-count string       MetaNode delete batch count
       --delete-worker-sleep-ms string   MetaNode delete worker sleep time with millisecond. if 0 for no sleep
   -h, --help                            help for delelerate
       --mark-delete-rate string         DataNode batch mark delete limit rate. if 0 for no infinity limit

Capacity Management

  1. What if the capacity of Volume is used-out?
$ cfs-cli volume expand {volume name} {capacity / GB}
  1. How to optimize the read/write performance from the Volume side？

The more dp that can be read and written, the better read and write performance of the Volume. ChubaoFS adopts a dynamic space allocation mechanism. After creating a Volume, it will pre-allocate a certain data partition dp for the Volume. When the number of dp that can be read and written is less than 10, the dp number will be automatically expanded by a step of 10. If you want to manually increase the number of readable and writable dp, you can use the following command:

$ cfs-cli volume create-dp {volume name} {number}

The default size of a dp is 120GB. Please add dp based on the actual cost to avoid overdrawing all dp.

  1. How to reclaim the excess space of Volume
$ cfs-cli volume shrink {volume name} {capacity / GB}

If the set value is less than %120 of the used amount, the operation fails.

  1. What if the cluster space is not enough?

Prepare the new dn and mn, start it by json configuration file with the current master hosts, and the dn/mn will be automatically added to the cluster.

Zone

Setting the zone can prevent the failure of a single zone witch causes the entire cluster unavailable. If parameter zoneName is set correctly, the node will automatically join the zone(cell).

  1. See zones list
$ cfs-cli zone list
  1. What if accidentally set the volume zone by mistake?
$ cfs-cli volume update {volume name}  --zone-name={zone name}
  1. What happens if MetaNde and DataNode do not set zone?

Most of the parameters in the cluster have default values, and the default zone name is default. Please note that there must be enough dn and mn in a zone at the same time, otherwise, creating a volume in the zone will either fail to initialize data partitions or initialize meta partitions.

  1. The meaning of NodeSet？

Each zone will have several NodeSets, and each NodeSet can carry 18 nodes by default. Because ChubaoFS implements multi-raft, each node starts a raft server process, and each raft server manages m raft instances on the node. If the other replication group members of these raft members are distributed on N nodes, then the raft heartbeat will be transmitted between N nodes. As the cluster scale expands, N will become larger. Through the NodeSet restriction, the heartbeat is relatively independent within the NodeSet, which avoids the heartbeat storm of the cluster dimension. We use the multi raft and nodeset mechanisms together to avoid the problem of the raft heartbeat storm.

nodeset
  1. How to distribute dp/mp of a volume in NodeSet?

dp/mp is evenly distributed in NodeSet. On dp/mp creating, it will locate on the NodeSet behind the NodeSet of last dp/mp overall.

  1. How to plan the amount of NodeSet？

For 3-replicas dp/mp, a dp/mp will select the ns only when there are at least 3 available nodes in ns. count (ns) >= 18 * n + 3

Node status is abnormal

View node status through cli tool

$ cfs-cli datanode list
[Data nodes]
ID        ADDRESS               WRITABLE    STATUS
7         192.168.0.31:17310    No          Inactive
8         192.168.0.32:17310    No          Inactive
9         192.168.0.33:17310    Yes         Active
10        192.168.0.35:17310    Yes         Active
11        192.168.0.34:17310    Yes         Active

$ cfs-cli metanode list
[Meta nodes]
ID        ADDRESS               WRITABLE    STATUS
2        192.168.0.21:17210    No          Inactive
3        192.168.0.22:17210    No          Inactive
4        192.168.0.23:17210    Yes         Active
5        192.168.0.25:17210    Yes         Active
6        192.168.0.24:17210    Yes         Active
  1. Reasons for Datanode WRITABLE=No
  • The node is waiting for the decommission to complete
  • Node disk is out of use
  • The node has just started and is restoring data locally
  1. Reasons for Metanode WRITABLE=No
  • The node is waiting for the decommission to complete
  • The node memory has reached totalMemory
  • The node has just started and is restoring data locally
  1. If one of the three masters in the cluster is broken, can the remaining two restarts provide services normally?

Yes. Since the Master uses the RAFT, it can provide services normally when the number of remaining nodes exceeds 50% of the total number of nodes.

  1. Reasons for STATUS=Inactive
  • The network connection between the node and the master is interrupted. You need to check the network status and restore the network connection
  • The node process hangs, you need to check whether the server process of the node is abnormally terminated, at this time restart the process to recover

Upgrade

  1. Steps
  1. Download and unzip the latest binary file compression package from ChubaoFS official website https://github.com/chubaofs/chubaofs/releases
  2. Freeze the cluster
$ cfs-cli cluster freeze true
  1. Confirm the startup configuration file, do not change important information such as the data directory and port in the configuration file
  2. Stop the old server process
  3. Start the new server process
  4. Check that the node status recovered healthy after the upgrade, IsActive: Active
$ cfs-cli datanode info 192.168.0.33:17310
[Data node info]
 ID                  : 9
 Address             : 192.168.0.33:17310
 Carry               : 0.06612836801123345
 Used ratio          : 0.0034684352702178426
 Used                : 96 GB
 Available           : 27 TB
 Total               : 27 TB
 Zone                : default
 IsActive            : Active
 Report time         : 2020-07-27 10:23:20
 Partition count     : 16
 Bad disks           : []
 Persist partitions  : [2 3 5 7 8 10 11 12 13 14 15 16 17 18 19 20]
  1. Upgrade the next node (in order to reduce the impact on the client, especially in a relatively large user volume, you need to upgrade the MetaNode nodes one by one), the upgrade sequence is shown bellow
upgrade
  1. After upgrading a master, it is found that the monitoring system is not displayed in time?

Check whether the configuration of this master node is correct, especially the id; check the master error log to find out whether there is a large number of no leader errors, and query the keyword leaderChange in the master warn log to check the reason for the leader change, and then check the raft warn log for further analysis.

  1. Can I modify the port number of the configuration during the upgrade?

No, ip+port constitutes the unique identifier of mn and dn instances, and will be treated as a new node after modification.

Update Configuration Online

  1. Update mn threshold
$ cfs-cli cluster set threshold { value }
  1. Update cluster configuration
  2. Update Volume configuration
$ cfs-cli volume set -h
Set configuration of the volume
Usage:
  cfs-cli volume set [VOLUME NAME] [flags]
Flags:
      --authenticate string    Enable authenticate
      --capacity uint          Specify volume capacity [Unit: GB]
      --enable-token string    ReadOnly/ReadWrite token validation for fuse client
      --follower-read string   Enable read form replica follower
  -h, --help                   help for set
      --replicas int           Specify volume replicas number
  -y, --yes                    Answer yes for all questions
      --zonename string        Specify volume zone name
  1. Update log level

An interface to update the log level of master, MetaNode, and DataNode online is provided:

$ http://127.0.0.1:{profPort}/loglevel/set?level={log-level}

Supported log-level: debug,info,warn,error,critical,read,write,fatal

Update Configuration Offline

  1. Update master IP

After the ip address of the three-node master is replaced, all mn, dn and other applications that reference the master ip address need to be restarted after modifying the configuration.

  1. Update DataNode/MetaNode port number

It is not recommended to modify the dn/mn port. Because dn/mn is registered through ip:port in the master. If the port is modified, the master will consider it as a brand new node, and the old node is in the state of Inactive.

  1. Update MetaNode totalMemory

totalMemory refers to the total memory size of MetaNode. When the memory usage of MetaNode is higher than this value, MetaNode becomes read-only. Usually this value is smaller than the node memory. If MetaNode and DataNode are deployed in a mixed deployment, extra memory needs to be reserved for the DataNode.

  1. Update DataNode reservedSpace

In the dn startup configuration json file, the number in the second half of the disk parameter is the value of Reserved Space, unit(byte).

{ ...
"disks": [
   "/cfs/disk:10737418240"
  ],
  ...
  }
  1. For more configurations, see Resource Manager (Master) Data Subsystem Meta SubsystemClient.

Handle Logs

  1. What to do if tens of GB of logs are generated every day, which takes up too much disk space?

In a production environment, you can set the log level to warn or error, which will significantly reduce the amount of logs.

$ http://127.0.0.1:{profPort}/loglevel/set?level={log-level}

Supported log-level: debug,info,warn,error,critical,read,write,fatal

  1. Datanode warn log
checkFileCrcTaskErr clusterID[xxx] partitionID:xxx File:xxx badCrc On xxx:

Analysis: The Master schedules the dn to check the crc data every few hours. This error indicates that the crc check failed and the file data is wrong. At this time, it is necessary to further analyze the error according to the partitionID and File in the warn message and with the assistance of the info log.

  1. Datanode error log
  2. Master error log
clusterID[xxx] addr[xxx]_op[xx] has no response util time out

Analysis:The response timed out when the Master sends the [Op] command to mn or dn, check the network between Master and mn/dn; check whether the dn/mn service process is alive.

  1. Master warn log
  2. Metanode error log
Error metaPartition(xx) changeLeader to (xx):

Analysis:Leader change, a normal action.

inode count is not equal, vol[xxx], mpID[xx]

Analysis:The number of inode is inconsistent. Because as long as two of the three copies are successful, the write is successful, so there will be inconsistencies in the three copies. Check the log for the specific reason.

  1. Metanode warn log
  2. Client warn log
operation.go:189: dcreate: packet(ReqID(151)Op(OpMetaCreateDentry)PartitionID(0)ResultCode(ExistErr)) mp(PartitionID(1) Start(0) End(16777216) Members([192.168.0.23:17210 192.168.0.24:17210 192.168.0.21:17210]) LeaderAddr(192.168.0.23:17210) Status(2)) req({ltptest 1 1 16777218 test.log 420}) result(ExistErr)

Analysis: ExistErr indicates that the file name already exists during the rename operation. It is an upper-level business operation problem. Maintainers can ignore this.

extent_handler.go:498: allocateExtent: failed to create extent, eh(ExtentHandler{ID(xxx)Inode(xxx)FileOffset(xxxx)StoreMode(1)}) err(createExtent: ResultCode NOK, packet(ReqID(xxxxx)Op(OpCreateExtent)Inode(0)FileOffset(0)Size(86)PartitionID(xxxxx)ExtentID(xx)ExtentOffset(0)CRC(0)ResultCode(IntraGroupNetErr)) datapartionHosts(1.1.0.0:17310) ResultCode(IntraGroupNetErr))

Analysis: The client sends a request to create an extent to an mp and fails, it will soon try to request another mp.

  1. Client error log
appendExtentKey: packet(%v) mp(%v) req(%v) result(NotExistErr)

Analysis: This error indicates that the file was deleted before written, which is an upper-level business operation problem. Maintainers can ignore this.

conn.go:103:sendToMetaPartition: retry failed req(ReqID(xxxx)Op(OpMetaInodeGet)PartitionID(0)ResultCode(Unknown ResultCode(0)))mp(PartitionID(xxxx) Start(xxx) End(xxx) Members([xxx xxxx xxxx]) LeaderAddr(xxxx) Status(2)) mc(partitionID(xxxx) addr(xxx)) err([conn.go 129] Failed to read from conn, req(ReqID(xxxx)Op(OpMetaInodeGet)PartitionID(0)ResultCode(Unknown ResultCode(0))) :: read tcp 10.196.0.10:42852->11.196.1.11:9021: i/o timeout) resp(<nil>)

Analysis 1:The network connection between the client and the MetaNode is abnormal. According to the error message “10.196.0.10:42852->11.196.1.11:9021”, check whether the network between the two addresses is normal

Analysis 2:Check if the MetaNode process hangs on “11.196.1.11:9021”

  1. Raft warn log
  2. Raft error log
raft.go:446: [ERROR] raft partitionID[1105] replicaID[6] not active peer["nodeID":"6","peerID":"0","priority":"0","type":"PeerNormal"]

Analysis: This is caused by excessive network pressure and increased delay. After the raft election interval is exceeded, the raft replication group loses the leader. After the network is restored, re-elect the leader, the error will disappear by itself.

Data Loss and Consistence

  1. All data of a single dn/mn is lost

This situation can be equivalent to a dn/mn failure. You can log off the node through decommission, and then restart the node to re-register the node to the Master, and the Master will treat it as a new member.

  1. Accidentally deleted file data in a dp directory in dn

dn has the function of automatically repairing data. If the data has not been repaired for a long time, you can manually restart the current dn process, which will trigger the data repair process.

Fuse Client

  1. Memory and performance optimization issues
  1. Mount issues
  • Does it support subdirectory mounting?

Yes. Set subdir in the configuration file

  • What are the reasons for the mount failure

    • If you see the following output,
    $ ... err(readFromProcess: sub-process: fusermount: exec: "fusermount": executable file not found in $PATH)

    Check if fuse is installed, if not, install it

    $ rpm –qa|grep fuse
  yum install fuse
    • Check if the mount directory exists
    • Check whether the mount point directory is empty
    • Check whether the mount point has been umount
    • Check whether the mount point status is normal. If the following message appears on the mount point mnt, you need to umount first, and then start the client
    $ ls -lih
ls: cannot access 'mnt': Transport endpoint is not connected
total 0
6443448706 drwxr-xr-x 2 root root 73 Jul 29 06:19 bin
 811671493 drwxr-xr-x 2 root root 43 Jul 29 06:19 conf
6444590114 drwxr-xr-x 3 root root 28 Jul 29 06:20 log
         ? d????????? ? ?    ?     ?            ? mnt
 540443904 drwxr-xr-x 2 root root 45 Jul 29 06:19 script
    • Check whether the configuration file is correct, master address, volume name and other information
    • If none of the above problems exist, locate the error through the client error log to see if it is the mount failure caused by the MetaNode or master service
  1. IO issues
  • IOPS is too high and the client’s memory usage exceeds 3GB or even higher. Is there any way to limit IOPS?

Limit the frequency of client response to io requests by modifying the client rate limit.

#see current iops:
$ http://[ClientIP]:[profPort]/rate/get
#set iops, default:-1(no limits)
$ http://[ClientIP]:[profPort]/rate/set?write=800&read=800

  • io delay too high for ls or other operations

    • Because the client reads and writes files through the http protocol, please check whether the network status is healthy
    • Check whether there is an overloaded mn, whether the mn process is hanging, you can restart mn, or expand a new mn to the cluster and take the mp on the overloaded mn decommission to relieve the pressure of mn
  1. Does ChubaoFS provide strong consistence guarantees?

No.ChubaoFS has relaxed POSIX consistency semantics, i.e., instead of providing strong consistency guarantees, it only ensures sequential consistency for file/directory operations, and does not have any leasing mechanism to prevent multiple clients writing to the same file/directory. It depends on the upperlevel application to maintain a more restrict consistency level if necessary.

  1. Is it feasible to kill the client to directly stop the client service

No. It is recommended to umount first. After umount, the client process will automatically stop.