Welcome to ChubaoFS(Chubao File System)¶
Introduction¶
ChubaoFS(Chubao File System) is a distributed fle system that is designed to natively support large scale container platforms.
High Level Architecture¶
ChubaoFS consists of a metadata subsystem, a data subsystem, and a resource manager, and can be accessed by different clients (as a set of application processes) hosted on the containers through different file system instances called volumes.
The metadata subsystem stores the file metadata, and consists of a set of meta nodes. Each meta node consists of a set of meta partitions.
The data subsystem stores the file contents, and consists of a set of data nodes. Each data node consists of a set of data partitions.
The volume is a logical concept in ChubaoFS and consists of one or multiple meta partitions and one or multiple data partitions. Each partition can only be assigned to a single volume. From a client’s perspective, the volume can be viewed as a file system instance that contains data accessible by the containers. A volume can be mounted to multiple containers so that files can be shared among different clients simultaneously, and needs to be created at the very beginning before the any file operation. A ChubaoFS cluster deployed at each data center can have hundreds of thousands of volumes, whose data sizes vary from a few gigabytes to several terabytes.
Generally speaking, the resource manager periodically communicates with the metadata subsystem and data subsystem to manage the meta nodes and data nodes, respectively. Each client periodically communicates with the resource manager to obtain the up-to-date view of the mounted volume. A file operation usually initiates the communications from the client to the corresponding meta node and data node directly, without the involvement of the resource manager. The updated view of the mounted volume, as well as the file metadata are usually cached at the client side to reduce the communication overhead.
Features¶
Scalable Meta Management¶
The metadata operations could make up as much as half of typical file system workloads. On our platform, this becomes even more important as there could be heavy accesses to the metadata of files by tens of thousands of clients simultaneously. A single node that stores the file metadata could easily become the performance bottleneck. As a result, we employ a distributed metadata subsystem to manage the file metadata. In this way, the metadata requests from different clients can be forwarded to different nodes, which improves the scalability of the entire system. The metadata subsystem can be considered as an in-memory datastore of the file metadata. It can have thousands of meta nodes, each of which can have hundreds of billions of meta partitions. Each meta partition on a meta node stores the file metadata in memory by maintaining a set of inodes and a set of dentries. We employ two b-trees called inodeTree and dentryTree for fast lookup of inodes and dentries in the memory. The inodeTree is indexed by the inode id, and the dentryTree is indexed by the dentry name and the parent inode id. We also maintain a range of the inode ids (denoted as start and end) stored on a meta partition for splitting.
General-Purpose Storage Engine¶
To reduce the storage cost, many applications and services are served from the same shared storage infrastructure (aka “multi-tenancy”). The workloads of different applications and services are mixed together, where the file size can vary from a few kilobytes to hundreds of gigabytes, and the files can be written in a sequential or random fashion. For example, the log files usually need to be written sequentially in the execution order of the code; some data analytics in the machine learning domain are based on the data stored on the underlying file system; and a database engine running on top of the file system can modify the stored data frequently. A dedicated file system needs to be able to serve for all these different workloads with excellent performance.
Strong Replication Consistency¶
E-commence venders who move their line of business applications to the cloud usually prefer strong consistency. For example, an image processing service may not want to provide the customer with an outdated image that does not match the product description. This can be easily achieved if there is only one copy of the file. But to ensure a distributed file system to continue operating properly in the event of machines failures, which can be caused by various reasons such as faulty hard drives, bad motherboards, etc, there are usually multiple replicas of the same file. As a result, in a desired file system, the data read from any of the replicas must be consistent with each other.
Relaxed POSIX Semantics and Metadata Atomicity¶
In a POSIX-compliant distributed file system, the behavior of serving multiple processes on multiple client nodes should be the same as the behavior of a local file system serving multiple processes on a single node with direct attached storage. ChubaoFS provides POSIX-compliant APIs. However, the POSIX consistency semantics, as well as the atomicity requirement between the inode and dentry of the same file, have been carefully relaxed in order to better align with the needs of applications and to improve the system performance.
Quick Start Guide¶
Building¶
Use following command to build server and client:
make build
If the build is successful, cfs-server and cfs-client will be found in directory build/bin
Deployment¶
Start Resource Manager (Master)¶
nohup ./cfs-server -c master.json &
Sample master.json is shown as follows,
{
"role": "master",
"ip": "192.168.31.173",
"port": "80",
"prof":"10088",
"id":"1",
"peers": "1:192.168.31.173:80,2:192.168.31.141:80,3:192.168.30.200:80",
"retainLogs":"20000",
"logDir": "/export/Logs/master",
"logLevel":"info",
"walDir":"/export/Data/master/raft",
"storeDir":"/export/Data/master/rocksdbstore",
"consulAddr": "http://consul.prometheus-cfs.local",
"exporterPort": 9510,
"clusterName":"cfs",
"metaNodeReservedMem": "134217728"
}
For detailed explanations of master.json, please refer to Resource Manager (Master).
Start Metanode¶
nohup ./cfs-server -c meta.json &
Sample meta.json is shown as follows,
{
"role": "metanode",
"listen": "9021",
"prof": "9092",
"logLevel": "info",
"metadataDir": "/export/Data/metanode",
"logDir": "/export/Logs/metanode",
"raftDir": "/export/Data/metanode/raft",
"raftHeartbeatPort": "9093",
"raftReplicaPort": "9094",
"totalMem": "17179869184",
"consulAddr": "http://consul.prometheus-cfs.local",
"exporterPort": 9511,
"masterAddr": [
"192.168.31.173:80",
"192.168.31.141:80",
"192.168.30.200:80"
]
}
For detailed explanations of meta.json, please refer to Meta Subsystem.
Start Datanode¶
Prepare data directories
Recommendation Using independent disks can reach better performance.
Disk preparation
1.1 Check available disks
fdisk -l
1.2 Build local Linux file system on the selected devices
mkfs.xfs -f /dev/sdx
1.3 Make mount point
mkdir /data0
1.4 Mount the device on mount point
mount /dev/sdx /data0
Start datanode
nohup ./cfs-server -c datanode.json &Sample datanode.json is shown as follows,
{ "role": "datanode", "port": "6000", "prof": "6001", "logDir": "/export/Logs/datanode", "raftDir": "/export/Data/datanode/raft", "logLevel": "info", "raftHeartbeat": "9095", "raftReplica": "9096", "consulAddr": "http://consul.prometheus-cfs.local", "exporterPort": 9512, "masterAddr": [ "192.168.31.173:80", "192.168.31.141:80", "192.168.30.200:80" ], "disks": [ "/data0:21474836480", "/data1:21474836480" ] }
For detailed explanations of datanode.json, please refer to Data Subsystem.
Start ObjectNode¶
nohup ./cfs-server -c objectnode.json &
Sample objectnode.json is shown as follows,
{
"role": "objectnode",
"domains": [
"object.cfs.local"
],
"listen": 80,
"masters": [
"master1.cfs.local:80",
"master2.cfs.local:80",
"master3.cfs.local:80"
],
"logLevel": "info",
"logDir": "/export/Logs/objectnode",
"region": "cn_bj"
}
For detailed explanations of meta.json, please refer to Object Subsystem (ObjectNode).
Create Volume¶
By default, there are only a few data partitions allocated upon volume creation, and will be dynamically expanded according to actual usage.
curl -v "http://192.168.31.173/admin/createVol?name=test&capacity=10000&owner=cfs"
For performance evaluation, extra data partitions shall be pre-created according to the amount of data nodes and disks to reach maximum performance.
curl -v "http://192.168.31.173/dataPartition/create?name=test&count=120"
Mount Client¶
Run
modprobe fuseto insert FUSE kernel module.Run
yum install -y fuseto install libfuse.Run
cfs-client -c fuse.jsonto start a client daemon.Sample fuse.json is shown as follows,
{ "mountPoint": "/mnt/fuse", "volName": "test", "owner": "cfs", "masterAddr": "192.168.31.173:80,192.168.31.141:80,192.168.30.200:80", "logDir": "/export/Logs/client", "profPort": "10094", "logLevel": "info" }
For detailed explanations of fuse.json, please refer to Client.
Note that end user can start more than one client on a single machine, as long as mountpoints are different.
Upgrading¶
- freeze the cluster
curl -v "http://192.168.31.173/cluster/freeze?enable=true"
- upgrade each module
- closed freeze flag
curl -v "http://192.168.31.173/cluster/freeze?enable=false"
Resource Manager (Master)¶
The resource manager manages the file system by processing different types of tasks, such as creating/deleting/updating/loading partitions and keeping track of the resource status (such as the memory/disk utilization). The resource manager is also responsible for creating new volumes and adding new meta/data nodes to the ChubaoFS cluster. It has multiple replicas, among which the consistency is maintained by a consensus algorithm such as Raft, and persisted to a key value store such as RocksDB for backup and recovery.
Utilization-Based Distribution/Placement¶
The resource manager is a utilization-based distribution strategy to places the file metadata and contents across different meta and data nodes. Because each node can have multiple partitions, and the partitions in a volume do not need to reside on the same node, this distribution can be controlled at a finer granularity, resulting a more efficient resource management. Specifically, the distribution of file metadata and contents works follows:
First, when mounting a volume, the client asks the resource manager for a set of available meta and data partitions. These partitions are usually the ones on the nodes with the lowest memory/disk utilizations. Later on, when writing a file, the client can only choose the meta and data partitions in a random fashion from the ones allocated by the resource manager.
Second, when the resource manager finds that all the partitions in a volume is about to be full, it automatically adds a set of new partitions to this volume. These partitions are usually the ones on the nodes with the lowest memory/disk utilizations. Note that, when a partition is full, or a threshold (i.e., the number of files on a meta partition or the number of extents on a data partition) is reached, no new data can be stored on this partition, although it can still be modified or deleted.
Replica Placement¶
When choosing partitions for the replicas, the resource manager ensures that two replicas of the same partition never reside on the same node.
Meta Partition Splitting¶
There is a special requirement when splitting a meta partition. In particular, if a meta partition is about to reach its upper limit of the number of stored inodes and dentries, a splitting task needs to be performed with the requirement to ensure that the inode ids stored at the newly created partition are unique from the ones stored at the original partition.
To meet this requirement, when splitting a meta partition, the resource manager cuts off the inode range of the meta partition in advance at a upper bound end, a value greater than highest inode id used so far (denoted as maxInodeID), and sends a split request to the meta node to (1) update the inode range from 1 to end for the original meta partition, and (2) create a new meta partition with the inode range from end + 1 to infinity for this volume. As a result, the inode range for these two meta partitions becomes [1, end] and [end + 1, infinity], respectively. If there is another file needs to be created, then its inode id will be chosen as maxInodeID + 1 in the original meta partition, or end + 1 in the newly created meta partition. The maxInodeID of each meta partition can be obtained by the periodical communication between the resource manager and the the meta nodes.
Exception Handling¶
When a request to a meta/data partition times out (e.g., due to network outage), the remaining replicas of this partition are marked as read-only. When a meta/data partition is no longer available (e.g., due to hardware failures), all the data on this partition will eventually be migrated to a new available partition manually. This unavailability is identified by the multiple failures reported by the node when operating the files.
Metadata Subsystem¶
The metadata operations could make up as much as half of typical file system workloads. this can be important as there could be heavy accesses to the metadata of files by tens of thousands of clients simultaneously. A single node that stores the file metadata could easily become the performance/storage bottleneck. As a result, we employ a distributed metadata subsystem to manage the file metadata. In this way, the metadata requests from different clients can be forwarded to different nodes, which improves the scalability of the entire system.
Internal Structure¶
The metadata subsystem can be considered as an in-memory datastore of the file metadata. It can have thousands of meta nodes, each of which can have a set of meta partitions. Each meta partition on a meta node stores the file metadata in memory by maintaining a set of inodes and a set of dentries.
Generally speaking, An inode is an object that represents the underlying file (or directory), and can be identified by an unsigned 64-bit integer called the inode id. A dentry is an object that represents the directory hierarchy and can be identified by a string name and the id of the parent inode. For example, if we have two directories foo and bar, where foo is the parent directory of bar, then there are two inodes: one for foo called i1, and the other for bar called i2, and one dentry to represent the hierarchy of these two directories where i2 is the current inode and i1 is the parent inode.
A meta partition can only store the inodes and dentries of the files from the same volume. We employ two b-trees called inodeTree and dentryTree for fast lookup of inodes and dentries in the memory. The inodeTree is indexed by the inode id, and the dentryTree is indexed by the dentry name and the parent inode id. We also maintain a range of the inode ids (denoted as start and end) stored on a meta partition for splitting (see Resource Manager (Master)).
Replication¶
The replication during file write is performed in terms of meta partitions. The replication consistency is ensured by a revision of the Raft consensus protocol called the MultiRaft, which has the advantage of reduced heartbeat network traffic comparing to the original version.
Failure Recovery¶
The in-memory meta partitions are persisted to the local disk by snapshots and logs for backup and recovery. Some techniques such as log compaction are used to reduce the log files sizes and shorten the recovery time.
It is worth noting that, a failure that happens during a metadata operation could result an orphan inode with which has no dentry to be associated. The memory and disk space occupied by this inode can be hard to free. To minimize the chance of this case to happen, the client always issues a retry after a failure until the request succeeds or the maximum retry limit is reached.
Data Subsystem¶
The data subsystem is optimized for the storage of large and small files, which can be accessed in a sequential or random fashion.
Features¶
- Large File Storage
For large files, the contents are stored as a sequence of one or multiple extents, which can be distributed across different data partitions on different data nodes. Writing a new file to the extent store always causes the data to be written at the zero-offset of a new extent, which eliminates the need for the offset within the extent. The last extent of a file does not need to fill up its size limit by padding (i.e., the extent does not have holes), and never stores the data from other files.
- Small File Storage
The contents of multiple small files are aggregated and stored in a single extent, and the physical offset of each file content in the extent is recorded in the corresponding meta node. ChubaoFS relies on the punch hole interface, textit{fallocate()}footnote{url{http://man7.org/linux/man-pages/man2/fallocate.2.html}}, to textit{asynchronous} free the disk space occupied by the to-be-deleted file. The advantage of this design is to eliminate the need of implementing a garbage collection mechanism and therefore avoid to employ a mapping from logical offset to physical offset in an extent~cite{haystack}. Note that this is different from deleting large files, where the extents of the file can be removed directly from the disk.
Replication
The replication is performed in terms of partitions during file writes. Depending on the file write pattern, ChubaoFS adopts different replication strategies.
When a file is sequentially written into ChubaoFS, a primary-backup replication protocol is used to ensure the strong consistency with optimized IO throughput.
When overwriting an existing file portion during random writes, we employ a MultiRaft-based replication protocol, which is similar to the one used in the metadata subsystem, to ensure the strong consistency.
Failure Recovery
Because of the existence of two different replication protocols, when a failure on a replica is discovered, we first start the recovery process in the primary-backup-based replication by checking the length of each extent and making all extents aligned. Once this processed is finished, we then start the recovery process in our MultiRaft-based replication.
HTTP APIs¶
| API | Method | Parameters | Description |
|---|---|---|---|
| /disks | GET | N/A | Get disk list and informations. |
| /partitions | GET | N/A | Get parttion list and infomartions. |
| /partition | GET | partitionId[int] | Get detail of specified partition. |
| /extent | GET | partitionId[int]&extentId[int] | Get extent informations. |
| /stats | GET | N/A | Get status of the datanode. |
Object Subsystem (ObjectNode)¶
The object subsystem provides S3-compatible object storage interface. It makes possible for ChubaoFS becomes a fusion storage that expose two type interface (POSIX and S3-compatible). So that user can operate files stored in ChubaoFS by using native Amazon S3 SDKs.
Structure¶
The ObjectNode is a functional subsystem node. It fetch volume view (volume topology) on demand from Resource Manager (Master). Each ObjectNode communicate with metadata subsystem (MetaNode) and data subsystem (DataNode) directly.
ObjectNode is stateless design with high scalability and it have ability to operate all files stored in the ChubaoFS cluster which it belong to directly without any volume-mount operation.
Features¶
- Provides object storage interface compatible with native Amazon S3 SDKs.
- Fusion storage expose two type interface (POSIX and S3-compatible).
- Stateless and high scalability
Semantic Transform¶
Based on our POSIX-compatible design. Every file operate request comes from object storage interface need to be made semantic transform to POSIX.
| POSIX | Object Storage |
|---|---|
Volume |
Bucket |
Path |
Key |
Example:
Put object ‘example/a/b.txt’ will be create and write data to file ‘/a/b.txt’ in volume ‘example’.
Authentication¶
The signature validation algorithm in object storage interface is fully compatible with native Amazon S3 service. The authentication consisting of AccessKey and SecretKey generated by Resource Manager (Master) with volume creation. The AccessKey is a 16-character string unique in the entire ChubaoFS cluster.
Authentication keys owned by volume and stored with volume view (volume topology) by Resource Manager (Master). User can fetch it by using administration API, see Get Volume Information at Volume
Invisible Temporary Data¶
In order to make write operation in object storage interface atomically. Every write operation will create and write data to an invisible temporary. The volume operator in ObjectNode puts file data into temporary which only have ‘inode’ without ‘dentry’ in metadata. When all the file data stored successfully, the volume operator create or update ‘dentry’ in metadata makes it visible to users.
Object Key Conflict (Important)¶
The POSIX and object storage are two different types of storage product, and the object storage is a Key-Value pair storage service. So that the object with key ‘a/b/c’ and the object with key ‘a/b’ are different object without any conflict.
But ChubaoFS is based on POSIX design. According to semantic transformation rule, the ‘b’ part in key ‘a/b/c’ transform to folder ‘b’ under the folder ‘a’ , and in key ‘a/b’ transform to file ‘b’ under the folder ‘a’.
The object key like this is conflict in ChuBaoFS.
Supported S3-Compatible APIs¶
Bucket APIs¶
| API | Reference |
|---|---|
HeadBucket |
https://docs.aws.amazon.com/AmazonS3/latest/API/API_HeadBucket.html |
GetBucketLocation |
https://docs.aws.amazon.com/AmazonS3/latest/API/API_GetBucketLocation.html |
Object APIs¶
Multipart Upload APIs¶
| API | Reference |
|---|---|
CreateMultipartUpload |
https://docs.aws.amazon.com/AmazonS3/latest/API/API_CreateMultipartUpload.html |
ListMultipartUploads |
https://docs.aws.amazon.com/AmazonS3/latest/API/API_ListMultipartUploads.html |
AbortMultipartUpload |
https://docs.aws.amazon.com/AmazonS3/latest/API/API_AbortMultipartUpload.html |
CompleteMultipartUpload |
https://docs.aws.amazon.com/AmazonS3/latest/API/API_CompleteMultipartUpload.html |
ListParts |
https://docs.aws.amazon.com/AmazonS3/latest/API/API_ListParts.html |
UploadPart |
https://docs.aws.amazon.com/AmazonS3/latest/API/API_UploadPart.html |
UploadPartCopy |
https://docs.aws.amazon.com/AmazonS3/latest/API/API_UploadPartCopy.html |
Supported SDKs¶
Object Node provides S3-compatible object storage interface, so that you can operate files by using native Amazon S3 SDKs.
| Name | Language | Link |
|---|---|---|
| AWS SDK for Java | Java |
https://aws.amazon.com/sdk-for-java/ |
| AWS SDK for JavaScript | JavaScript |
https://aws.amazon.com/sdk-for-browser/ |
| AWS SDK for JavaScript in Node.js | JavaScript |
https://aws.amazon.com/sdk-for-node-js/ |
| AWS SDK for Go | Go |
https://docs.aws.amazon.com/sdk-for-go/ |
| AWS SDK for PHP | PHP |
https://aws.amazon.com/sdk-for-php/ |
| AWS SDK for Ruby | Ruby |
https://aws.amazon.com/sdk-for-ruby/ |
| AWS SDK for .NET | .NET |
https://aws.amazon.com/sdk-for-net/ |
| AWS SDK for C++ | C++ |
https://aws.amazon.com/sdk-for-cpp/ |
| Boto3 | Python |
http://boto.cloudhackers.com |
Client¶
The client runs on each container executing application code and exposes a file system interface to applications and can access a mounted volume via a user-space file system interface such as FUSE.
Client Caching¶
The client process runs entirely in the user space with its own cache, which has been used in the following cases.
To reduce the communication with the resource manager, the client caches the addresses of the available meta and data partitions assigned to the mounted volume, which can be obtained at the startup, and periodically synchronizes this available partitions with the resource manager.
To reduce the communication with the meta nodes, the client also caches the returned inodes and dentries when creating new files, as well as the data partition id, the extent id and the offset, after the file has been written to the data node successfully. When a file is opened for read/write, the client will force the cache metadata to be synchronous with the meta node.
To reduce the communication with the data nodes, the client caches the most recently identified leader. Our observation is that, when reading a file, the client may not know which data node is the current leader because the leader could change after a failure recovery. As a result, the client may try to send the read request to each replica one by one until a leader is identified. However, since the leader does not change frequently, by caching the last identified leader, the client can have minimized number of retries in most cases.
Integration with FUSE¶
The ChubaoFS client has been integrated with FUSE to provide a file system interface in the user space. In the past, low performance is considered the main disadvantage of such user-space file systems. But over the years, FUSE has made several improvement on its performance such as multithreading and write-back cache. In the future, we plan to develop our own POSIX-compliant file system interface in the kernel space to completely eliminate the overhead from FUSE.
Currently the write-back cache feature does not work well in ChubaoFS due to the following reason. The default write behavior of FUSE is called directIO, which bypasses the kernel’s page cache. This results in performance problems on writing small files as each write pushes the file data to the user daemon. The solution FUSE implemented was to make the page cache support a write-back policy that aggregates small data first, and then make writes asynchronous. With that change, file data can be pushed to the user daemon in larger blobs at a time. However, in real production, we found that the write-back cache is not very useful, because a write operation usually invoke another process that tries to balance the dirty pages (pages in the main memory that have been modified during writing data to disk are marked as “dirty” and have to be flushed to disk before they can be freed), which incurs extra overhead. This overhead becomes more obvious when small files are continuously written through FUSE.
Authnode¶
Internet and Intranet are insecure places where malicious hackers usually use tools to “sniff” sensitive information off of the network. Worse yet, client/server are not trusted to be honest about their identities. Therefore, ChubaoFS encounters some common security problems once deployed in the Network.
Security Problems¶
- Unauthorized parties may access the resource, such as restful API, volume information.
- Communication channel is vulnerable to Man-in-the-middle (MITM) attacks.
High Level Architecture¶
Authnode is the security node providing a general Authentication & Authorization framework for ChubaoFS. Besides, Authnode acts as a centralized key store of both symmetric and asymmetric key. Authnode adopts and customizes the idea of authentication from Kerberos which is built on top of tickets. Specifically, whenever a client node (Master, Meta, Data or Client node) accesses a service, it’s firstly required to show the shared secret key for authentication in Authnode. If successful, Authnode would issue a time-limited ticket specifically for that service. With the purpose of authorization, capabilities are embedded in tickets to indicate who can do what on what resource.
In the context of Authnode, we define a node as Client if it is responsible to initialize a service request while Server or Service is defined as the node responding that request. In this case, any ChubaoFS nodes can be acted as either Client or Server.
The communication between Client and Server is based on HTTPS or TCP and the workflow of Authnode is depicted in the graph above and briefly described as follows:
Ticket Request (F1)¶
Before any service requests, client holding a secret key Ckey is required to get a ticket from Authnode for the service (called target service).
- C->A: Client sends a request including Client ID (id) and Service ID (sid) indicating target service.
- C<-A: Server lookups client secret key (CKey) and server secret key (Skey) from key store and responds a Ckey-encrypted message mainly including and a session key (sess_key) and target service ticket (ticket) encrypted with secret key Skey of target service if verification succeeds.
After obtaining the ticket and processing some security checks with Ckey, client has in its possession sess_key and Skey{ticket} for future service requests.
Service Request in HTTPS (F2)¶
If a service request is sent via HTTPS protocol, it has the following steps:
- C->S: Client sends a request containing SKey {ticket}.
- C<-S: Server (1) performs message decryption and get the ticket, (2) verifies its capabilities and (3) responds the data which is encrypted with sess_key extracted from ticket.
Client uses sess_key to decrypt message returned from server and verify its validity.
Service Request in TCP (F3)¶
If a service request is sent via TCP protocol, it has the following steps:
- C->S: Client sends a request containing SKey {ticket}.
- C<-S: Server (1) decrypts the ticket and validate its capabilities, (2) extracts sess_key, (3) generates a random number s_n and (4) responds this number encrypted with sess_key.
- C->S: Client decrypts the replied message and send to server another message including a randomly generated number s_c and s_n + 1, both of which are encrypted with sess_key.
- C<-S: Server verifies whether s_n has been increased by one and sends back a message including s_c + 1 which is encrypted by sess_key if the verification succeeds.
- C<->S: Client verifies whether s_c has been increased by one after message decryption. If successful, an authenticated communication channel has been established between client and server. Based on this channel, client and server can perform further communications.
Future Work¶
Authnode supports a general authentication and authorization which is an emergent need for ChubaoFS. There are two directions of security enhancements for ChubaoFS in the future.
Feature Enrichment¶
Current implementation of Authnode doesn’t support some advanced features:
- Key rotation: Shared secret keys are hardcoded in client and server and would not changed. It increases security risks that attacks break the encryption and find the keys. Rotating keys on a regular basis would help to mitigate such risks.
- Ticket revocation: For performance considerations, ticket would be valid for a while (such as several hours). If a client unfortunately leaks its ticket, malicious parties are able to use the ticket for service request during the period of being valid. Ticket revocation mechanism can prevent such an issue by revoking it once leakage happens.
- HSM support: Authnode is the security bottleneck in ChubaoFS. Breaking Authnode means compromising the whole system since it manages the key store. Hardware Security Module or HSM provides physical safeguards for key management. Having Authnode protected by HSM (for example SGX) can mitigate the risk of Authnode being compromised.
End-to-End Data Encryption¶
Current implementation of Authnode doesn’t systematically support encryption for data in transit and at rest even though we may use session key to encrypt data during communication. A more secure way to protect data is to have End-to-End Data Encryption. In particular, encryption keys are managed and distributed by Authnode and data are encrypted in client node, sent via network and stored in server. Compared with server side encryption based on existing tools (fscrypt, ecryptfs and dm-crypt), End-to-End Data Encryption has the following advantages at least:
- It mitigates data leakage once data servers (for example Data Node) are broken into by attackers since keys of data decoding are stored in Authnode.
- It provides a centralized management (rotation, revocation and generation) for encryption key.
Resource Manager (Master)¶
The cluster contains dataNodes,metaNodes,vols,dataPartitions and metaPartitions,they are managed by master server. The master server caches the metadata in mem,persist to GoLevelDB,and ensure consistence by raft protocol.
The master server manages dataPartition id to dataNode server mapping,metaPartition id to metaNode server mapping.
At lease 3 master nodes are required in respect to high availability.
Features¶
- Multi-tenant, Resource Isolation
- dataNodes,metaNodes shared,vol owns dataPartition and metaPartition exclusive
- Asynchronous communication with dataNode and metaNode
Configurations¶
ChubaoFS use JSON as configuration file format.
| Key | Type | Description | Mandatory |
|---|---|---|---|
| role | string | Role of process and must be set to master | Yes |
| ip | string | host ip | Yes |
| port | string | Http port which api service listen on | Yes |
| prof | string | golang pprof port | Yes |
| id | string | identy different master node | Yes |
| peers | string | the member information of raft group | Yes |
| logDir | string | Path for log file storage | Yes |
| logLevel | string | Level operation for logging. Default is error. | No |
| retainLogs | string | the number of raft logs will be retain. | Yes |
| walDir | string | Path for raft log file storage. | Yes |
| storeDir | string | Path for RocksDB file storage,path must be exist | Yes |
| clusterName | string | The cluster identifier | Yes |
| exporterPort | int | The prometheus exporter port | No |
| consulAddr | string | The consul register addr for prometheus exporter | No |
| metaNodeReservedMem | string | If the metanode memory is below this value, it will be marked as read-only. |
Example:
{
"role": "master",
"id":"1",
"ip": "192.168.31.173",
"port": "80",
"prof":"10088",
"peers": "1:192.168.31.173:80,2:192.168.31.141:80,3:192.168.30.200:80",
"retainLogs":"20000",
"logDir": "/export/Logs/master",
"logLevel":"info",
"walDir":"/export/Data/master/raft",
"storeDir":"/export/Data/master/rocksdbstore",
"exporterPort": 9510,
"consulAddr": "http://consul.prometheus-cfs.local",
"clusterName":"test",
"metaNodeReservedMem": "134217728"
}
Start Service¶
nohup ./master -c config.json > nohup.out &
Meta Subsystem¶
Metanode is the manager of meta partitions and replicated by MultiRaft. Each metanode manages various of partitions. Each partition covers an inode range, and maintains two in-memory btrees: inode btree and dentry btree.
At lease 3 meta nodes are required in respect to high availability.
| Key | Type | Description | Mandatory | |
|---|---|---|---|---|
| role | string | Role of process and must be set to metanode | Yes | |
| listen | string | Listen and accept port of the server | Yes | |
| prof | string | Pprof port | Yes | |
| localIP | string | IP of network to be choose | No,If not specified, the ip address used to communicate with the master is used. | |
| logLevel | string | Level operation for logging. Default is error | No | |
| metadataDir | string | MetaNode store snapshot directory” | Yes | |
| logDir | string | Log directory | Yes | |
| raftDir | string | Raft wal directory | Yes | |
| raftHeartbeatPort | string | Raft heartbeat port | Yes | |
| raftReplicaPort | string | Raft replicate port | Yes | |
| consulAddr | string | Addresses of monitor system | No | |
| exporterPort | string | Port for monitor system | No | |
| masterAddr | string | Addresses of master server | Yes | |
| totalMem | string | Max memory metadata used | No |
Example:
{
"role": "metanode",
"listen": "9021",
"prof": "9092",
"localIP":"192.168.31.173",
"logLevel": "debug",
"metadataDir": "/export/Data/metanode",
"logDir": "/export/Logs/metanode",
"raftDir": "/export/Data/metanode/raft",
"raftHeartbeatPort": "9093",
"raftReplicaPort": "9094",
"consulAddr": "http://consul.prometheus-cfs.local",
"exporterPort": 9511,
"totalMem": "17179869184",
"masterAddr": [
"192.168.31.173:80",
"192.168.31.141:80",
"192.168.30.200:80"
]
}
Data Subsystem¶
How To Start DataNode¶
Start a DataNode process by execute the server binary of ChubaoFS you built with -c argument and specify configuration file. At least 4 data nodes are required in respect to high availability.
nohup cfs-server -c datanode.json &
Configurations¶
| Key | Type | Description | Mandatory |
|---|---|---|---|
| role | string | Role of process and must be set to datanode | Yes |
| port | string | Port of TCP network to be listen | Yes |
| localIP | string | IP of network to be choose | No,If not specified, the ip address used to communicate with the master is used. |
| prof | string | Port of HTTP based prof and api service | Yes |
| logDir | string | Path for log file storage | Yes |
| logLevel | string | Level operation for logging. Default is error | No |
| raftHeartbeat | string | Port of raft heartbeat TCP network to be listen | Yes |
| raftReplica | string | Port of raft replicate TCP network to be listen | Yes |
| raftDir | string | Path for raft log file storage | No |
| consulAddr | string | Addresses of monitor system | No |
| exporterPort | string | Port for monitor system | No |
| masterAddr | string slice | Addresses of master server | Yes |
| disks | string slice | Format: PATH:RETAIN.
PATH: Disk mount point. RETAIN: Retain space. (Ranges: 20G-50G.)
|
Yes |
Example:
{
"role": "datanode",
"port": "6000",
"prof": "6001",
"localIP":"192.168.31.174",
"logDir": "/export/Logs/datanode",
"logLevel": "info",
"raftHeartbeat": "9095",
"raftReplica": "9096",
"raftDir": "/export/Data/datanode/raft",
"consulAddr": "http://consul.prometheus-cfs.local",
"exporterPort": 9512,
"masterAddr": [
"192.168.31.173:80",
"192.168.31.141:80",
"192.168.30.200:80"
],
"disks": [
"/data0:21474836480",
"/data1:21474836480"
]
}
Object Subsystem (ObjectNode)¶
How To Provide Object Storage Service with Object Subsystem (ObjectNode)¶
Start a ObjectNode process by execute the server binary of ChubaoFS you built with -c argument and specify configuration file.
nohup cfs-server -c objectnode.json &
Configurations¶
Object Node using JSON format configuration file.
Properties
| Key | Type | Description | Mandatory |
|---|---|---|---|
| role | string | Role of process and must be set to objectnode | Yes |
| listen | string | Listen and accept port of the server. Default: 80 | Yes |
| region | string | Region of this gateway. Used by S3-like interface signature validation. Default: cfs_default | No |
| domains | string slice | Format: DOMAIN.
DOMAIN: Domain of S3-like interface which makes wildcard domain support
|
No |
| logDir | string | Log directory | Yes |
| logLevel | string | Level operation for logging. Default is error | No |
| masters | string slice | Format: HOST:PORT.
HOST: Hostname, domain or IP address of master (resource manager).
PORT: port number which listened by this master
|
Yes |
| exporterPort | string | Port for monitor system | No |
| prof | string | Pprof port | Yes |
Example:
{
"role": "objectnode",
"listen": 80,
"region": "test",
"domains": [
"object.cfs.local"
],
"logDir": "/opt/cfs/objectnode/logs",
"logLevel": "debug",
"masters": [
"172.20.240.95:7002",
"172.20.240.94:7002",
"172.20.240.67:7002"
],
"exporterPort": 9512,
"prof": "7013"
}
Fetch Authentication Keys¶
Authentication keys owned by volume and stored with volume view (volume topology) by Resource Manager (Master). User can fetch it by using administration API, see Get Volume Information at Volume
Using Object Storage Interface¶
Object Subsystem (ObjectNode) provides S3-compatible object storage interface, so that you can operate files by using native Amazon S3 SDKs.
For detail about list of supported APIs, see Supported S3-compatible APIs at Object Subsystem (ObjectNode)
For detail about list of supported SDKs, see Supported SDKs at Object Subsystem (ObjectNode)
Client¶
Prepare Config File¶
fuse.json
{
"mountPoint": "/mnt/fuse",
"volName": "test",
"owner": "cfs",
"masterAddr": "192.168.31.173:80,192.168.31.141:80,192.168.30.200:80",
"logDir": "/export/Logs/client",
"logLevel": "info",
"profPort": "10094"
}
| Name | Type | Description | Mandatory |
|---|---|---|---|
| mountPoint | string | Mount point | Yes |
| volName | string | Volume name | Yes |
| owner | string | Owner name as authentication | Yes |
| masterAddr | string | Resource manager IP address | Yes |
| logDir | string | Path to store log files | No |
| logLevel | string | Log level：debug, info, warn, error | No |
| profPort | string | Golang pprof port | No |
| exporterPort | string | Performance monitor port | No |
| consulAddr | string | Performance monitor server address | No |
| lookupValid | string | Lookup valid duration in FUSE kernel module, unit: sec | No |
| attrValid | string | Attr valid duration in FUSE kernel module, unit: sec | No |
| icacheTimeout | string | Inode cache valid duration in client | No |
| enSyncWrite | string | Enable DirectIO sync write, i.e. make sure data is fsynced in data node | No |
| autoInvalData | string | Use AutoInvalData FUSE mount option | No |
Mount¶
Use the example fuse.json, and client is mounted on the directory /mnt/fuse. All operations to /mnt/fuse would be performed on the backing distributed file system.
./cfs-client -c fuse.json
Authnode¶
Authnode provides a general authentication & authorization service among ChubaoFS nodes. Client, Master, Meta and Data node are required to be authenticated and authorized before any resource access in another node.
Initially, each node (Auth, Client, Master, Meta or Data node) is launched with a secure key which is distributed by a authenticated person (for instance, cluster admin). With a valid key, a node can be identified in Authnode service and granted a ticket for resource access.
The overall work flow is: key creation and distribution –> ticket retrieval with key –> resource access with ticket.
Concepts¶
- Key: a bit of secret shared data between a node and Authnode that asserts identity of a client.
- Ticket: a bit of data that cryptographically asserts identity and authorization (through a list of capabilities) for a service for a period of time.
- Capability: a capability is defined in the format of node:object:action where node refers to a service node (such as auth, master, meta or data), object refers to the resource and action refers to permitted activities (such as read, write and access). See examples below.
| Capability | Specifications |
|---|---|
| auth:createkey:access | Access permission for createkey in Authnode |
| master:*:* | Any permissions for any objects in Master node |
| *:*:* | Any permissions for any objects in any nodes |
Client Tool¶
cfs-authtool is a client-side utility of Authnode, providing key managements (creation, view and modification) and ticket retrieval in and from Authnode keystore. In particular, Each key is associated with an entity name or id, secret key string, creation time, role and capability specification.
| Key | Type | Description |
|---|---|---|
| id | string | Unique key identifier composed of letters and digits |
| key | string | Base64 encoded secret key |
| role | string | The role of the key (either client or service) |
| caps | string | The capabilities of the key |
Build¶
Use the following commands to build client side tool for Authnode:
$ git clone http://github.com/chubaofs/chubaofs.git
$ cd chubaofs
$ make build
If successful, the tool cfs-authtool can be found in build/bin.
Synopsis¶
cfs-authtool ticket -host=AuthNodeAddress [-keyfile=Keyfile] [-output=TicketOutput] [-https=TrueOrFalse -certfile=AuthNodeCertfile] TicketService Service
cfs-authtool api -host=AuthNodeAddress -ticketfile=TicketFile [-data=RequestDataFile] [-output=KeyFile] [-https=TrueOrFalse -certfile=AuthNodeCertfile] Service Request
cfs-authtool authkey [-keylen=KeyLength]
TicketService := [getticket]
Service := [AuthService | MasterService | MetaService | DataService]
Request := [createkey | deletekey | getkey | addcaps | deletecaps | getcaps | addraftnode | removeraftnode]
AuthNode Configurations¶
Authnode use JSON as configuration file format.
| Key | Type | Description | Mandatory |
|---|---|---|---|
| role | string | Role of process and must be set to master | Yes |
| ip | string | host ip | Yes |
| port | string | Http port which api service listen on | Yes |
| prof | string | golang pprof port | Yes |
| id | string | identy different master node | Yes |
| peers | string | the member information of raft group | Yes |
| logDir | string | Path for log file storage | Yes |
| logLevel | string | Level operation for logging. Default is error. | No |
| retainLogs | string | the number of raft logs will be retain. | Yes |
| walDir | string | Path for raft log file storage. | Yes |
| storeDir | string | Path for RocksDB file storage,path must be exist | Yes |
| clusterName | string | The cluster identifier | Yes |
| exporterPort | int | The prometheus exporter port | No |
| authServiceKey | string | The secret key used for authentication of AuthNode | Yes |
| authRootKey | string | The secret key used for key derivation (session and client secret key) | Yes |
| enableHTTPS | bool | Option whether enable HTTPS protocol | No |
Example:
{ "role": "authnode", "ip": "192.168.0.14", "port": "8080", "prof":"10088", "id":"1", "peers": "1:192.168.0.14:8080,2:192.168.0.15:8081,3:192.168.0.16:8082", "logDir": "/export/Logs/authnode", "logLevel":"info", "retainLogs":"100", "walDir":"/export/Data/authnode/raft", "storeDir":"/export/Data/authnode/rocksdbstore", "exporterPort": 9510, "consulAddr": "http://consul.prometheus-cfs.local", "clusterName":"test", "authServiceKey":"9h/sNq4+5CUAyCnAZM927Y/gubgmSixh5hpsYQzZG20=", "authRootKey":"wbpvIcHT/bLxLNZhfo5IhuNtdnw1n8kom+TimS2jpzs=", "enableHTTPS":false }
Steps for Starting ChubaoFS with AuthNode¶
Create Authnode key¶
Run the command:
$ ./cfs-authtool authkey
If successful, two key files can be generated authroot.json and authservice.json under current directory.
They represent authServiceKey and authRootKey respectively.
example authservice.json :
{ "id": "AuthService", "key": "9h/sNq4+5CUAyCnAZM927Y/gubgmSixh5hpsYQzZG20=", "create_ts": 1573801212, "role": "AuthService", "caps": "{\"*\"}" }
Edit authnode.json in docker/conf as following:
authRootKey: use the value ofkeyinauthroot.jsonauthServiceKey: use the value ofkeyinauthService.json
example authnode.json :
{ "role": "authnode", "ip": "192.168.0.14", "port": "8080", "prof":"10088", "id":"1", "peers": "1:192.168.0.14:8080,2:192.168.0.15:8081,3:192.168.0.16:8082", "retainLogs":"2", "logDir": "/export/Logs/authnode", "logLevel":"info", "walDir":"/export/Data/authnode/raft", "storeDir":"/export/Data/authnode/rocksdbstore", "exporterPort": 9510, "consulAddr": "http://consul.prometheus-cfs.local", "clusterName":"test", "authServiceKey":"9h/sNq4+5CUAyCnAZM927Y/gubgmSixh5hpsYQzZG20=", "authRootKey":"wbpvIcHT/bLxLNZhfo5IhuNtdnw1n8kom+TimS2jpzs=", "enableHTTPS":false }
Start Authnode Cluster¶
In directory docker/authnode, run the following command to start a Authnode cluster.
$ docker-compose up -d
Create admin in Authnode¶
Get Authnode ticket using authServiceKey:
$ ./cfs-authtool ticket -host=192.168.0.14:8080 -keyfile=authservice.json -output=ticket_auth.json getticket AuthService
example ticket_auth.json :
{ "id": "AuthService", "session_key": "A9CSOGEN9CFYhnFnGwSMd4WFDBVbGmRNjaqGOhOinJE=", "service_id": "AuthService", "ticket": "RDzEiRLX1xjoUyp2TDFviE/eQzXGlPO83siNJ3QguUrtpwiHIA3PLv4edyKzZdKcEb3wikni8UxBoIJRhKzS00+nB7/9CjRToAJdT9Glhr24RyzoN8psBAk82KEDWJhnl+Y785Av3f8CkNpKv+kvNjYVnNKxs7f3x+Ze7glCPlQjyGSxqARyLisoXoXbiE6gXR1KRT44u7ENKcUjWZ2ZqKEBML9U4h0o58d3IWT+n4atWKtfaIdp6zBIqnInq0iUueRzrRlFEhzyrvi0vErw+iU8w3oPXgTi+um/PpUyto20c1NQ3XbnkWZb/1ccx4U0" }
Create admin using Authnode ticket:
$ ./cfs-authtool api -host=192.168.0.14:8080 -ticketfile=ticket_auth.json -data=data_admin.json -output=key_admin.json AuthService createkey
example data_admin.json :
{ "id": "admin", "role": "service", "caps": "{\"API\":[\"*:*:*\"]}" }
Create key for ChubaoFS cluster¶
- Get Authnode ticket using admin key:
$ ./cfs-authtool ticket -host=192.168.0.14:8080 -keyfile=key_admin.json -output=ticket_admin.json getticket AuthService
- Create key for Master
$ ./cfs-authtool api -host=192.168.0.14:8080 -ticketfile=ticket_admin.json -data=data_master.json -output=key_master.json AuthService createkey
example data_master.json :
{ "id": "MasterService", "role": "service", "caps": "{\"API\":[\"*:*:*\"]}" }
- Specifications:
id: will set Client ID
role: will set the role of id
caps: will set the capabilities of id
- Edit
master.jsonas following:
masterServiceKey: use the value ofkeyinkey_master.json
Create key for Client
$ ./cfs-authtool api -host=192.168.0.14:8080 -ticketfile=ticket_admin.json -data=data_client.json -output=key_client.json AuthService createkey
example
data_client.json:{ "id": "ltptest", "role": "client", "caps": "{\"API\":[\"*:*:*\"], \"Vol\":[\"*:*:*\"]}" }
- Edit
client.jsonas following: clientKey: use the value ofkeyinkey_client.json
example
client.json：{ "masterAddr": "192.168.0.11:17010,192.168.0.12:17010,192.168.0.13:17010", "mountPoint": "/cfs/mnt", "volName": "ltptest", "owner": "ltptest", "logDir": "/cfs/log", "logLevel": "info", "consulAddr": "http://192.168.0.100:8500", "exporterPort": 9500, "profPort": "17410", "authenticate": true, "ticketHost": "192.168.0.14:8080,192.168.0.15:8081,192.168.0.16:8082", "clientKey": "jgBGSNQp6mLbu7snU8wKIdEkytzl+pO5/OZOJPpIgH4=", "enableHTTPS": "false" }
- Edit
- Specifications:
authenticate: will enable authentication flow if set true.
ticketHost: will set the IP/URL of Authnode cluster.
clientKey: will set the key generated by Authnode
enableHTTPS: will enable HTTPS if set true.
Start ChubaoFS cluster¶
Run the following to launch ChubaoFS cluster with AuthNode enabled:
$ docker/run_docker.sh -r -d /data/disk
Generate Certificate¶
To prevent MITM (Man In The Middle) attacks, HTTPS is required for the communication between client and service. The following steps show the generation of self-sign a certificate with a private (.key) and public key.
- Generating Key and Self Signed Cert:
$ openssl req \
-x509 \
-nodes \
-newkey rsa:2048 \
-keyout server.key \
-out server.crt \
-days 3650 \
-subj "/C=GB/ST=China/L=Beijing/O=jd.com/OU=Infra/CN=*"
- `server.crt`: `AuthNode` public certificate needed to be sent to `Client`
- `server.key`: `AuthNode` private key needed to be securely placed in `/app` folder in `Authnode`
For easy deployment, current implementation of AuthNode uses TLS option insecure_skip_verify and tls.RequireAndVerifyClientCert, which would skip secure verification of both client and server. For environment with high security command, these options should be turned off.
Monitor¶
ChubaoFS use prometheus as metrics collector. It simply config as follow in master，metanode，datanode，client’s config file：
{
"exporterPort": 9510,
"consulAddr": "http://consul.prometheus-cfs.local"
}
- exporterPort：prometheus exporter Port. when set，can export prometheus metrics from URL(http://$hostip:$exporterPort/metrics). If not set, prometheus exporter will unavailable；
- consulAddr: consul register address，it can work with prometheus to auto discover deployed chubaofs nodes, if not set, consul register will not work.
Using grafana as prometheus metrics web front：
Also, user can use prometheus alertmanager to capture and route alerts to the correct receiver. please visit prometheus alertmanger web-doc
Grafana DashBoard Config:
{
"annotations": {
"list": [
{
"builtIn": 1,
"datasource": "-- Grafana --",
"enable": true,
"hide": true,
"iconColor": "rgba(0, 211, 255, 1)",
"name": "Annotations & Alerts",
"type": "dashboard"
}
]
},
"editable": true,
"gnetId": null,
"graphTooltip": 0,
"id": 21,
"iteration": 1573200227128,
"links": [],
"panels": [
{
"collapsed": false,
"gridPos": {
"h": 1,
"w": 24,
"x": 0,
"y": 0
},
"id": 40,
"panels": [],
"title": "Cluster",
"type": "row"
},
{
"cacheTimeout": null,
"colorBackground": false,
"colorValue": false,
"colors": [
"#299c46",
"rgba(237, 129, 40, 0.89)",
"#d44a3a"
],
"datasource": null,
"format": "none",
"gauge": {
"maxValue": 100,
"minValue": 0,
"show": false,
"thresholdLabels": false,
"thresholdMarkers": true
},
"gridPos": {
"h": 4,
"w": 4,
"x": 0,
"y": 1
},
"id": 38,
"interval": null,
"links": [],
"mappingType": 1,
"mappingTypes": [
{
"name": "value to text",
"value": 1
},
{
"name": "range to text",
"value": 2
}
],
"maxDataPoints": 100,
"nullPointMode": "connected",
"nullText": null,
"postfix": "",
"postfixFontSize": "50%",
"prefix": "",
"prefixFontSize": "50%",
"rangeMaps": [
{
"from": "null",
"text": "N/A",
"to": "null"
}
],
"sparkline": {
"fillColor": "rgba(31, 118, 189, 0.18)",
"full": false,
"lineColor": "rgb(31, 120, 193)",
"show": true
},
"tableColumn": "",
"targets": [
{
"expr": "count(go_info{cluster=~\"$cluster\", app=~\"$app\", role=~\"master\"})",
"format": "time_series",
"intervalFactor": 1,
"refId": "A"
}
],
"thresholds": "",
"title": "master_count",
"type": "singlestat",
"valueFontSize": "80%",
"valueMaps": [
{
"op": "=",
"text": "N/A",
"value": "null"
}
],
"valueName": "current"
},
{
"cacheTimeout": null,
"colorBackground": false,
"colorValue": false,
"colors": [
"#299c46",
"rgba(237, 129, 40, 0.89)",
"#d44a3a"
],
"datasource": null,
"format": "none",
"gauge": {
"maxValue": 100,
"minValue": 0,
"show": false,
"thresholdLabels": false,
"thresholdMarkers": true
},
"gridPos": {
"h": 4,
"w": 4,
"x": 4,
"y": 1
},
"id": 42,
"interval": null,
"links": [],
"mappingType": 1,
"mappingTypes": [
{
"name": "value to text",
"value": 1
},
{
"name": "range to text",
"value": 2
}
],
"maxDataPoints": 100,
"nullPointMode": "connected",
"nullText": null,
"postfix": "",
"postfixFontSize": "50%",
"prefix": "",
"prefixFontSize": "50%",
"rangeMaps": [
{
"from": "null",
"text": "N/A",
"to": "null"
}
],
"sparkline": {
"fillColor": "rgba(31, 118, 189, 0.18)",
"full": false,
"lineColor": "rgb(31, 120, 193)",
"show": true
},
"tableColumn": "",
"targets": [
{
"expr": "count(go_info{cluster=~\"$cluster\", app=~\"$app\", role=~\"metanode\"})",
"format": "time_series",
"intervalFactor": 1,
"refId": "A"
}
],
"thresholds": "",
"title": "metanode_count",
"type": "singlestat",
"valueFontSize": "80%",
"valueMaps": [
{
"op": "=",
"text": "N/A",
"value": "null"
}
],
"valueName": "current"
},
{
"cacheTimeout": null,
"colorBackground": false,
"colorValue": false,
"colors": [
"#299c46",
"rgba(237, 129, 40, 0.89)",
"#d44a3a"
],
"datasource": "default",
"format": "none",
"gauge": {
"maxValue": 100,
"minValue": 0,
"show": false,
"thresholdLabels": false,
"thresholdMarkers": true
},
"gridPos": {
"h": 4,
"w": 4,
"x": 8,
"y": 1
},
"id": 41,
"interval": null,
"links": [],
"mappingType": 1,
"mappingTypes": [
{
"name": "value to text",
"value": 1
},
{
"name": "range to text",
"value": 2
}
],
"maxDataPoints": 100,
"nullPointMode": "connected",
"nullText": null,
"postfix": "",
"postfixFontSize": "50%",
"prefix": "",
"prefixFontSize": "50%",
"rangeMaps": [
{
"from": "null",
"text": "N/A",
"to": "null"
}
],
"sparkline": {
"fillColor": "rgba(31, 118, 189, 0.18)",
"full": false,
"lineColor": "rgb(31, 120, 193)",
"show": true
},
"tableColumn": "",
"targets": [
{
"expr": "count(go_info{cluster=\"$cluster\", app=\"$app\", role=\"dataNode\"})",
"format": "time_series",
"intervalFactor": 1,
"refId": "A"
}
],
"thresholds": "",
"title": "datanode_count",
"type": "singlestat",
"valueFontSize": "80%",
"valueMaps": [
{
"op": "=",
"text": "N/A",
"value": "null"
}
],
"valueName": "current"
},
{
"cacheTimeout": null,
"colorBackground": false,
"colorValue": false,
"colors": [
"#299c46",
"rgba(237, 129, 40, 0.89)",
"#d44a3a"
],
"datasource": "default",
"format": "none",
"gauge": {
"maxValue": 100,
"minValue": 0,
"show": false,
"thresholdLabels": false,
"thresholdMarkers": true
},
"gridPos": {
"h": 4,
"w": 3,
"x": 12,
"y": 1
},
"id": 86,
"interval": null,
"links": [],
"mappingType": 1,
"mappingTypes": [
{
"name": "value to text",
"value": 1
},
{
"name": "range to text",
"value": 2
}
],
"maxDataPoints": 100,
"nullPointMode": "connected",
"nullText": null,
"postfix": "",
"postfixFontSize": "50%",
"prefix": "",
"prefixFontSize": "50%",
"rangeMaps": [
{
"from": "null",
"text": "N/A",
"to": "null"
}
],
"sparkline": {
"fillColor": "rgba(31, 118, 189, 0.18)",
"full": false,
"lineColor": "rgb(31, 120, 193)",
"show": true
},
"tableColumn": "",
"targets": [
{
"expr": "count(go_info{cluster=~\"$cluster\", app=~\"$app\", role=~\"fuseclient\"})",
"format": "time_series",
"intervalFactor": 1,
"refId": "A"
}
],
"thresholds": "",
"title": "client_count",
"type": "singlestat",
"valueFontSize": "80%",
"valueMaps": [
{
"op": "=",
"text": "N/A",
"value": "null"
}
],
"valueName": "current"
},
{
"cacheTimeout": null,
"colorBackground": false,
"colorValue": false,
"colors": [
"#299c46",
"rgba(237, 129, 40, 0.89)",
"#d44a3a"
],
"datasource": "default",
"format": "none",
"gauge": {
"maxValue": 100,
"minValue": 0,
"show": false,
"thresholdLabels": false,
"thresholdMarkers": true
},
"gridPos": {
"h": 4,
"w": 3,
"x": 15,
"y": 1
},
"id": 93,
"interval": null,
"links": [],
"mappingType": 1,
"mappingTypes": [
{
"name": "value to text",
"value": 1
},
{
"name": "range to text",
"value": 2
}
],
"maxDataPoints": 100,
"nullPointMode": "connected",
"nullText": null,
"postfix": "",
"postfixFontSize": "50%",
"prefix": "",
"prefixFontSize": "50%",
"rangeMaps": [
{
"from": "null",
"text": "N/A",
"to": "null"
}
],
"sparkline": {
"fillColor": "rgba(31, 118, 189, 0.18)",
"full": false,
"lineColor": "rgb(31, 120, 193)",
"show": true
},
"tableColumn": "",
"targets": [
{
"expr": "sum([[app]]_master_vol_count{cluster=~\"$cluster\"})",
"format": "time_series",
"intervalFactor": 1,
"refId": "A"
}
],
"thresholds": "",
"title": "vol_count",
"type": "singlestat",
"valueFontSize": "80%",
"valueMaps": [
{
"op": "=",
"text": "N/A",
"value": "null"
}
],
"valueName": "current"
},
{
"cacheTimeout": null,
"colorBackground": false,
"colorValue": false,
"colors": [
"#299c46",
"rgba(237, 129, 40, 0.89)",
"#d44a3a"
],
"datasource": null,
"decimals": null,
"format": "dateTimeFromNow",
"gauge": {
"maxValue": 100,
"minValue": 0,
"show": false,
"thresholdLabels": false,
"thresholdMarkers": true
},
"gridPos": {
"h": 4,
"w": 3,
"x": 18,
"y": 1
},
"id": 113,
"interval": null,
"links": [],
"mappingType": 1,
"mappingTypes": [
{
"name": "value to text",
"value": 1
},
{
"name": "range to text",
"value": 2
}
],
"maxDataPoints": 100,
"nullPointMode": "connected",
"nullText": null,
"postfix": "",
"postfixFontSize": "50%",
"prefix": "",
"prefixFontSize": "50%",
"rangeMaps": [
{
"from": "null",
"text": "N/A",
"to": "null"
}
],
"sparkline": {
"fillColor": "rgba(31, 118, 189, 0.18)",
"full": false,
"lineColor": "rgb(31, 120, 193)",
"show": true
},
"tableColumn": "",
"targets": [
{
"expr": "process_start_time_seconds{cluster=\"$cluster\", app=\"$app\", instance=~\"$instance\"}*1000",
"format": "time_series",
"intervalFactor": 1,
"refId": "A"
}
],
"thresholds": "",
"title": "process_start_time",
"type": "singlestat",
"valueFontSize": "80%",
"valueMaps": [
{
"op": "=",
"text": "N/A",
"value": "null"
}
],
"valueName": "current"
},
{
"aliasColors": {},
"bars": false,
"dashLength": 10,
"dashes": false,
"datasource": null,
"fill": 1,
"gridPos": {
"h": 6,
"w": 7,
"x": 0,
"y": 5
},
"id": 115,
"legend": {
"avg": false,
"current": false,
"max": false,
"min": false,
"show": true,
"total": false,
"values": false
},
"lines": true,
"linewidth": 1,
"links": [],
"nullPointMode": "null",
"percentage": false,
"pointradius": 5,
"points": false,
"renderer": "flot",
"seriesOverrides": [],
"spaceLength": 10,
"stack": false,
"steppedLine": false,
"targets": [
{
"expr": "sum(go_info{app=\"$app\", cluster=\"$cluster\"}) by (cluster, app, role)",
"format": "time_series",
"intervalFactor": 1,
"legendFormat": "{{cluster}}-{{role}}",
"refId": "A"
},
{
"expr": "count(go_info{cluster=~\"$cluster\", app=~\"$app\", role=~\"$role\"})",
"format": "time_series",
"hide": true,
"intervalFactor": 1,
"legendFormat": "[[role]]-count",
"refId": "B"
}
],
"thresholds": [],
"timeFrom": null,
"timeShift": null,
"title": "node alive",
"tooltip": {
"shared": true,
"sort": 0,
"value_type": "individual"
},
"type": "graph",
"xaxis": {
"buckets": null,
"mode": "time",
"name": null,
"show": true,
"values": []
},
"yaxes": [
{
"format": "locale",
"label": null,
"logBase": 1,
"max": null,
"min": null,
"show": true
},
{
"format": "short",
"label": null,
"logBase": 1,
"max": null,
"min": null,
"show": true
}
],
"yaxis": {
"align": false,
"alignLevel": null
}
},
{
"aliasColors": {},
"bars": false,
"dashLength": 10,
"dashes": false,
"datasource": null,
"fill": 1,
"gridPos": {
"h": 6,
"w": 7,
"x": 7,
"y": 5
},
"id": 177,
"legend": {
"avg": false,
"current": false,
"max": false,
"min": false,
"show": true,
"total": false,
"values": false
},
"lines": true,
"linewidth": 1,
"links": [],
"nullPointMode": "null",
"percentage": false,
"pointradius": 5,
"points": false,
"renderer": "flot",
"seriesOverrides": [
{
"alias": "/used_ratio.*/",
"yaxis": 2
}
],
"spaceLength": 10,
"stack": false,
"steppedLine": false,
"targets": [
{
"expr": "cfs_master_dataNodes_total_GB{app=\"$app\",cluster=\"$cluster\"}>0",
"format": "time_series",
"hide": false,
"intervalFactor": 1,
"legendFormat": "total_GB",
"refId": "B"
},
{
"expr": "cfs_master_dataNodes_used_GB{app=\"$app\",cluster=\"$cluster\"}>0",
"format": "time_series",
"hide": false,
"intervalFactor": 1,
"legendFormat": "used_GB",
"refId": "A"
},
{
"expr": "cfs_master_dataNodes_increased_GB{app=\"$app\",cluster=\"$cluster\"}>0",
"format": "time_series",
"hide": false,
"intervalFactor": 1,
"legendFormat": "increased_GB",
"refId": "C"
},
{
"expr": "sum(cfs_master_dataNodes_used_GB{app=\"$app\",cluster=\"$cluster\"}) / sum(cfs_master_dataNodes_total_GB{app=\"$app\",cluster=\"$cluster\"})",
"format": "time_series",
"hide": false,
"intervalFactor": 1,
"legendFormat": "used_ratio",
"refId": "D"
}
],
"thresholds": [],
"timeFrom": null,
"timeShift": null,
"title": "dataNode_size",
"tooltip": {
"shared": true,
"sort": 0,
"value_type": "individual"
},
"type": "graph",
"xaxis": {
"buckets": null,
"mode": "time",
"name": null,
"show": true,
"values": []
},
"yaxes": [
{
"format": "decgbytes",
"label": null,
"logBase": 1,
"max": null,
"min": null,
"show": true
},
{
"format": "percentunit",
"label": null,
"logBase": 1,
"max": null,
"min": null,
"show": true
}
],
"yaxis": {
"align": false,
"alignLevel": null
}
},
{
"aliasColors": {},
"bars": false,
"dashLength": 10,
"dashes": false,
"datasource": null,
"fill": 1,
"gridPos": {
"h": 6,
"w": 7,
"x": 14,
"y": 5
},
"id": 92,
"legend": {
"avg": false,
"current": false,
"max": false,
"min": false,
"show": true,
"total": false,
"values": false
},
"lines": true,
"linewidth": 1,
"links": [],
"nullPointMode": "null",
"percentage": false,
"pointradius": 5,
"points": false,
"renderer": "flot",
"seriesOverrides": [],
"spaceLength": 10,
"stack": false,
"steppedLine": false,
"targets": [
{
"expr": "sum([[app]]_master_metaNodes_total_GB{cluster=~\"$cluster\"})",
"format": "time_series",
"intervalFactor": 1,
"legendFormat": "total_GB",
"refId": "B"
},
{
"expr": "sum([[app]]_master_metaNodes_used_GB{cluster=~\"$cluster\"})",
"format": "time_series",
"intervalFactor": 1,
"legendFormat": "used_GB",
"refId": "A"
},
{
"expr": "sum([[app]]_master_metaNodes_increased_GB{cluster=~\"$cluster\"})",
"format": "time_series",
"intervalFactor": 1,
"legendFormat": "increased_GB",
"refId": "C"
}
],
"thresholds": [],
"timeFrom": null,
"timeShift": null,
"title": "metanode_size",
"tooltip": {
"shared": true,
"sort": 0,
"value_type": "individual"
},
"type": "graph",
"xaxis": {
"buckets": null,
"mode": "time",
"name": null,
"show": true,
"values": []
},
"yaxes": [
{
"format": "decgbytes",
"label": null,
"logBase": 1,
"max": null,
"min": null,
"show": true
},
{
"format": "short",
"label": null,
"logBase": 1,
"max": null,
"min": null,
"show": true
}
],
"yaxis": {
"align": false,
"alignLevel": null
}
},
{
"aliasColors": {},
"bars": false,
"dashLength": 10,
"dashes": false,
"datasource": null,
"fill": 1,
"gridPos": {
"h": 6,
"w": 7,
"x": 0,
"y": 11
},
"id": 175,
"legend": {
"avg": false,
"current": false,
"max": false,
"min": false,
"show": true,
"total": false,
"values": false
},
"lines": true,
"linewidth": 1,
"links": [],
"nullPointMode": "null",
"percentage": false,
"pointradius": 5,
"points": false,
"renderer": "flot",
"seriesOverrides": [
{
"alias": "/.*_ratio.*/",
"yaxis": 2
}
],
"spaceLength": 10,
"stack": false,
"steppedLine": false,
"targets": [
{
"expr": "cfs_master_vol_usage_ratio{app=\"$app\",cluster=\"$cluster\"}",
"format": "time_series",
"hide": true,
"intervalFactor": 1,
"legendFormat": "{{volName}}_usage_ratio",
"refId": "B"
},
{
"expr": "cfs_master_vol_total_GB{app=\"$app\",cluster=\"$cluster\"}",
"format": "time_series",
"hide": false,
"intervalFactor": 1,
"legendFormat": "{{volName}}_total_GB",
"refId": "A"
},
{
"expr": "cfs_master_vol_used_GB{app=\"$app\",cluster=\"$cluster\"}",
"format": "time_series",
"hide": false,
"intervalFactor": 1,
"legendFormat": "{{volName}}_used_GB",
"refId": "C"
},
{
"expr": "cfs_master_vol_used_GB{app=\"$app\",cluster=\"$cluster\"} / cfs_master_vol_total_GB{app=\"$app\",cluster=\"$cluster\"} ",
"format": "time_series",
"hide": false,
"intervalFactor": 1,
"legendFormat": "{{volName}}_used_ratio",
"refId": "D"
}
],
"thresholds": [],
"timeFrom": null,
"timeShift": null,
"title": "vol_size",
"tooltip": {
"shared": true,
"sort": 0,
"value_type": "individual"
},
"type": "graph",
"xaxis": {
"buckets": null,
"mode": "time",
"name": null,
"show": true,
"values": []
},
"yaxes": [
{
"format": "decgbytes",
"label": null,
"logBase": 1,
"max": null,
"min": null,
"show": true
},
{
"format": "percentunit",
"label": null,
"logBase": 1,
"max": null,
"min": null,
"show": true
}
],
"yaxis": {
"align": false,
"alignLevel": null
}
},
{
"aliasColors": {},
"bars": false,
"dashLength": 10,
"dashes": false,
"datasource": "default",
"fill": 1,
"gridPos": {
"h": 6,
"w": 7,
"x": 7,
"y": 11
},
"id": 90,
"legend": {
"avg": false,
"current": false,
"max": false,
"min": false,
"show": true,
"total": false,
"values": false
},
"lines": true,
"linewidth": 1,
"links": [],
"nullPointMode": "null",
"percentage": false,
"pointradius": 5,
"points": false,
"renderer": "flot",
"seriesOverrides": [],
"spaceLength": 10,
"stack": false,
"steppedLine": false,
"targets": [
{
"expr": "sum([[app]]_master_vol_count{cluster=~\"$cluster\"})",
"format": "time_series",
"intervalFactor": 1,
"legendFormat": "vol_count",
"refId": "M"
}
],
"thresholds": [],
"timeFrom": null,
"timeShift": null,
"title": "vol_count",
"tooltip": {
"shared": true,
"sort": 0,
"value_type": "individual"
},
"type": "graph",
"xaxis": {
"buckets": null,
"mode": "time",
"name": null,
"show": true,
"values": []
},
"yaxes": [
{
"format": "locale",
"label": null,
"logBase": 1,
"max": null,
"min": null,
"show": true
},
{
"format": "short",
"label": null,
"logBase": 1,
"max": null,
"min": null,
"show": true
}
],
"yaxis": {
"align": false,
"alignLevel": null
}
},
{
"aliasColors": {},
"bars": false,
"dashLength": 10,
"dashes": false,
"datasource": "default",
"fill": 1,
"gridPos": {
"h": 6,
"w": 7,
"x": 14,
"y": 11
},
"id": 73,
"legend": {
"avg": false,
"current": false,
"max": false,
"min": false,
"show": true,
"total": false,
"values": false
},
"lines": true,
"linewidth": 1,
"links": [],
"nullPointMode": "null",
"percentage": false,
"pointradius": 5,
"points": false,
"renderer": "flot",
"seriesOverrides": [
{
"alias": "/.*_QPS/",
"yaxis": 2
}
],
"spaceLength": 10,
"stack": false,
"steppedLine": false,
"targets": [
{
"expr": "sum([[app]]_metanode_OpMetaOpen{cluster=~\"$cluster\"})",
"format": "time_series",
"intervalFactor": 1,
"legendFormat": "Open",
"refId": "A"
},
{
"expr": "sum([[app]]_metanode_OpMetaLookup{cluster=~\"$cluster\"})",
"format": "time_series",
"intervalFactor": 1,
"legendFormat": "Lookup",
"refId": "B"
},
{
"expr": "sum([[app]]_metanode_OpMetaNodeHeartbeat{cluster=~\"$cluster\"})",
"format": "time_series",
"intervalFactor": 1,
"legendFormat": "NodeHeartbeat",
"refId": "C"
},
{
"expr": "sum([[app]]_metanode_OpMetaReadDir{cluster=~\"$cluster\"})",
"format": "time_series",
"intervalFactor": 1,
"legendFormat": "ReadDir",
"refId": "D"
},
{
"expr": "sum([[app]]_metanode_OpMetaReleaseOpen{cluster=~\"$cluster\"})",
"format": "time_series",
"intervalFactor": 1,
"legendFormat": "ReleaseOpen",
"refId": "E"
},
{
"expr": "sum([[app]]_metanode_OpMetaSetattr{cluster=~\"$cluster\"})",
"format": "time_series",
"intervalFactor": 1,
"legendFormat": "Setattr",
"refId": "F"
},
{
"expr": "sum([[app]]_metanode_OpMetaTruncate{cluster=~\"$cluster\"})",
"format": "time_series",
"intervalFactor": 1,
"legendFormat": "Truncate",
"refId": "G"
},
{
"expr": "sum(rate([[app]]_metanode_OpMetaOpen{cluster=~\"$cluster\"}[1m]))",
"format": "time_series",
"intervalFactor": 1,
"legendFormat": "Open_QPS",
"refId": "H"
},
{
"expr": "sum(rate([[app]]_metanode_OpMetaLookup{cluster=~\"$cluster\"}[1m]))",
"format": "time_series",
"intervalFactor": 1,
"legendFormat": "Lookup_QPS",
"refId": "I"
},
{
"expr": "sum(rate([[app]]_metanode_OpMetaReadDir{cluster=~\"$cluster\"}[1m]))",
"format": "time_series",
"intervalFactor": 1,
"legendFormat": "ReadDir_QPS",
"refId": "K"
},
{
"expr": "sum(rate([[app]]_metanode_OpMetaTruncate{cluster=~\"$cluster\"}[1m]))",
"format": "time_series",
"intervalFactor": 1,
"legendFormat": "Truncate_QPS",
"refId": "L"
},
{
"expr": "sum(rate([[app]]_metanode_OpMetaReleaseOpen{cluster=~\"$cluster\"}[1m]))",
"format": "time_series",
"intervalFactor": 1,
"legendFormat": "ReleaseOpen_QPS",
"refId": "J"
},
{
"expr": "sum(rate([[app]]cfs_metanode_OpMetaSetattr{cluster=~\"$cluster\"}[1m]))",
"format": "time_series",
"intervalFactor": 1,
"legendFormat": "Setattr_QPS",
"refId": "M"
}
],
"thresholds": [],
"timeFrom": null,
"timeShift": null,
"title": "metanode_OpMeta",
"tooltip": {
"shared": true,
"sort": 0,
"value_type": "individual"
},
"type": "graph",
"xaxis": {
"buckets": null,
"mode": "time",
"name": null,
"show": true,
"values": []
},
"yaxes": [
{
"format": "ns",
"label": null,
"logBase": 1,
"max": null,
"min": null,
"show": true
},
{
"format": "ops",
"label": null,
"logBase": 1,
"max": null,
"min": null,
"show": true
}
],
"yaxis": {
"align": false,
"alignLevel": null
}
},
{
"aliasColors": {},
"bars": false,
"dashLength": 10,
"dashes": false,
"datasource": "default",
"fill": 1,
"gridPos": {
"h": 6,
"w": 7,
"x": 0,
"y": 17
},
"id": 179,
"legend": {
"avg": false,
"current": false,
"max": false,
"min": false,
"show": true,
"total": false,
"values": false
},
"lines": true,
"linewidth": 1,
"links": [],
"nullPointMode": "null",
"percentage": false,
"pointradius": 5,
"points": false,
"renderer": "flot",
"seriesOverrides": [
{
"alias": "sum"
}
],
"spaceLength": 10,
"stack": false,
"steppedLine": false,
"targets": [
{
"expr": "sum(cfs_master_disk_error{app=\"$app\", cluster=\"$cluster\"} > 0)",
"format": "time_series",
"intervalFactor": 1,
"legendFormat": "sum",
"refId": "D"
},
{
"expr": "cfs_master_disk_error{app=\"$app\", cluster=\"$cluster\"} > 0",
"format": "time_series",
"intervalFactor": 1,
"legendFormat": "{{addr}}_{{path}}",
"refId": "A"
}
],
"thresholds": [],
"timeFrom": null,
"timeShift": null,
"title": "disk_error",
"tooltip": {
"shared": true,
"sort": 0,
"value_type": "individual"
},
"type": "graph",
"xaxis": {
"buckets": null,
"mode": "time",
"name": null,
"show": true,
"values": []
},
"yaxes": [
{
"format": "locale",
"label": null,
"logBase": 1,
"max": null,
"min": null,
"show": true
},
{
"format": "locale",
"label": null,
"logBase": 1,
"max": null,
"min": null,
"show": true
}
],
"yaxis": {
"align": false,
"alignLevel": null
}
},
{
"aliasColors": {},
"bars": false,
"dashLength": 10,
"dashes": false,
"datasource": "default",
"fill": 1,
"gridPos": {
"h": 6,
"w": 7,
"x": 7,
"y": 17
},
"id": 180,
"legend": {
"avg": false,
"current": false,
"max": false,
"min": false,
"show": true,
"total": false,
"values": false
},
"lines": true,
"linewidth": 1,
"links": [],
"nullPointMode": "null",
"percentage": false,
"pointradius": 5,
"points": false,
"renderer": "flot",
"seriesOverrides": [],
"spaceLength": 10,
"stack": false,
"steppedLine": false,
"targets": [
{
"expr": "cfs_master_metaNodes_inactive{app=\"$app\", cluster=\"$cluster\"}> 0",
"format": "time_series",
"hide": false,
"intervalFactor": 1,
"legendFormat": "{{instance}}",
"refId": "B"
},
{
"expr": "sum(cfs_master_metaNodes_inactive{app=\"$app\", cluster=\"$cluster\"}> 0)",
"format": "time_series",
"hide": false,
"intervalFactor": 1,
"legendFormat": "sum",
"refId": "A"
}
],
"thresholds": [],
"timeFrom": null,
"timeShift": null,
"title": "metaNodes_inactive",
"tooltip": {
"shared": true,
"sort": 0,
"value_type": "individual"
},
"type": "graph",
"xaxis": {
"buckets": null,
"mode": "time",
"name": null,
"show": true,
"values": []
},
"yaxes": [
{
"format": "locale",
"label": null,
"logBase": 1,
"max": null,
"min": null,
"show": true
},
{
"format": "short",
"label": null,
"logBase": 1,
"max": null,
"min": null,
"show": true
}
],
"yaxis": {
"align": false,
"alignLevel": null
}
},
{
"aliasColors": {},
"bars": false,
"dashLength": 10,
"dashes": false,
"datasource": "default",
"fill": 1,
"gridPos": {
"h": 6,
"w": 7,
"x": 14,
"y": 17
},
"id": 181,
"legend": {
"avg": false,
"current": false,
"max": false,
"min": false,
"show": true,
"total": false,
"values": false
},
"lines": true,
"linewidth": 1,
"links": [],
"nullPointMode": "null",
"percentage": false,
"pointradius": 5,
"points": false,
"renderer": "flot",
"seriesOverrides": [],
"spaceLength": 10,
"stack": false,
"steppedLine": false,
"targets": [
{
"expr": "cfs_master_dataNodes_inactive{app=\"$app\", cluster=\"$cluster\"}> 0",
"format": "time_series",
"hide": false,
"intervalFactor": 1,
"legendFormat": "{{instance}}",
"refId": "C"
},
{
"expr": "sum(cfs_master_dataNodes_inactive{app=\"$app\", cluster=\"$cluster\"}> 0)",
"format": "time_series",
"hide": false,
"intervalFactor": 1,
"legendFormat": "sum",
"refId": "A"
}
],
"thresholds": [],
"timeFrom": null,
"timeShift": null,
"title": "dataNodes_inactive",
"tooltip": {
"shared": true,
"sort": 0,
"value_type": "individual"
},
"type": "graph",
"xaxis": {
"buckets": null,
"mode": "time",
"name": null,
"show": true,
"values": []
},
"yaxes": [
{
"format": "locale",
"label": null,
"logBase": 1,
"max": null,
"min": null,
"show": true
},
{
"format": "short",
"label": null,
"logBase": 1,
"max": null,
"min": null,
"show": true
}
],
"yaxis": {
"align": false,
"alignLevel": null
}
},
{
"aliasColors": {},
"bars": false,
"dashLength": 10,
"dashes": false,
"datasource": "default",
"fill": 1,
"gridPos": {
"h": 6,
"w": 7,
"x": 0,
"y": 23
},
"id": 71,
"legend": {
"avg": false,
"current": false,
"max": false,
"min": false,
"show": true,
"total": false,
"values": false
},
"lines": true,
"linewidth": 1,
"links": [],
"nullPointMode": "null",
"percentage": false,
"pointradius": 5,
"points": false,
"renderer": "flot",
"seriesOverrides": [
{
"alias": "/.*_QPS/",
"yaxis": 2
}
],
"spaceLength": 10,
"stack": false,
"steppedLine": false,
"targets": [
{
"expr": "sum([[app]]_metanode_OpMetaBatchInodeGet{cluster=~\"$cluster\", instance=\"$instance\"})",
"format": "time_series",
"intervalFactor": 1,
"legendFormat": "BatchGet",
"refId": "A"
},
{
"expr": "sum([[app]]_metanode_OpMetaCreateInode{cluster=~\"$cluster\", instance=\"$instance\"})",
"format": "time_series",
"intervalFactor": 1,
"legendFormat": "Create",
"refId": "B"
},
{
"expr": "sum([[app]]_metanode_OpMetaDeleteInode{cluster=~\"$cluster\"})",
"format": "time_series",
"intervalFactor": 1,
"legendFormat": "Delete",
"refId": "C"
},
{
"expr": "sum([[app]]_metanode_OpMetaEvictInode{cluster=~\"$cluster\"})",
"format": "time_series",
"intervalFactor": 1,
"legendFormat": "Evict",
"refId": "D"
},
{
"expr": "sum([[app]]_metanode_OpMetaInodeGet{cluster=~\"$cluster\"})",
"format": "time_series",
"intervalFactor": 1,
"legendFormat": "GetInode",
"refId": "E"
},
{
"expr": "sum([[app]]_metanode_OpMetaLinkInode{cluster=~\"$cluster\"})",
"format": "time_series",
"intervalFactor": 1,
"legendFormat": "LinkInode",
"refId": "F"
},
{
"expr": "sum(rate([[app]]_metanode_OpMetaCreateInode{cluster=~\"$cluster\"}[1m]))",
"format": "time_series",
"intervalFactor": 1,
"legendFormat": "CreateInode_QPS",
"refId": "G"
},
{
"expr": "sum(rate([[app]]_metanode_OpMetaDeleteInode{cluster=~\"$cluster\"}[1m]))",
"format": "time_series",
"intervalFactor": 1,
"legendFormat": "DeleteInode_QPS",
"refId": "H"
},
{
"expr": "sum(rate([[app]]_metanode_OpMetaEvictInode{cluster=~\"$cluster\"}[1m]))",
"format": "time_series",
"intervalFactor": 1,
"legendFormat": "EvictInode_QPS",
"refId": "I"
},
{
"expr": "sum(rate([[app]]_metanode_OpMetaInodeGet{cluster=~\"$cluster\"}[1m]))",
"format": "time_series",
"intervalFactor": 1,
"legendFormat": "GetInode_QPS",
"refId": "J"
},
{
"expr": "sum(rate([[app]]_metanode_OpMetaLinkInode{cluster=~\"$cluster\"}[1m]))",
"format": "time_series",
"intervalFactor": 1,
"legendFormat": "LinkInode_QPS",
"refId": "K"
},
{
"expr": "sum(rate([[app]]_metanode_OpBatchInodeGet{cluster=~\"$cluster\"}[1m]))",
"format": "time_series",
"intervalFactor": 1,
"legendFormat": "BantchInodeGet_QPS",
"refId": "L"
},
{
"expr": "sum([[app]]_metanode_OpMetaTruncate{cluster=~\"$cluster\"})",
"format": "time_series",
"intervalFactor": 1,
"legendFormat": "TruncateInode",
"refId": "M"
},
{
"expr": "sum(rate([[app]]_metanode_OpMetaTruncate{cluster=~\"$cluster\"}[1m]))",
"format": "time_series",
"intervalFactor": 1,
"legendFormat": "TruncateInode_QPS",
"refId": "N"
},
{
"expr": "sum([[app]]_metanode_OpMetaUnlinkInode{cluster=~\"$cluster\", instance=\"$instance\"})",
"format": "time_series",
"intervalFactor": 1,
"legendFormat": "UnlinkInode",
"refId": "O"
},
{
"expr": "sum(rate([[app]]_metanode_OpMetaUnlinkInode{cluster=~\"$cluster\"}[1m]))",
"format": "time_series",
"intervalFactor": 1,
"legendFormat": "UnlinkInnode_QPS",
"refId": "P"
}
],
"thresholds": [],
"timeFrom": null,
"timeShift": null,
"title": "metanode_OpMetaInode",
"tooltip": {
"shared": true,
"sort": 0,
"value_type": "individual"
},
"type": "graph",
"xaxis": {
"buckets": null,
"mode": "time",
"name": null,
"show": true,
"values": []
},
"yaxes": [
{
"format": "ns",
"label": null,
"logBase": 1,
"max": null,
"min": null,
"show": true
},
{
"format": "ops",
"label": null,
"logBase": 1,
"max": null,
"min": null,
"show": true
}
],
"yaxis": {
"align": false,
"alignLevel": null
}
},
{
"aliasColors": {},
"bars": false,
"dashLength": 10,
"dashes": false,
"datasource": null,
"fill": 1,
"gridPos": {
"h": 6,
"w": 7,
"x": 7,
"y": 23
},
"id": 45,
"legend": {
"avg": false,
"current": false,
"max": false,
"min": false,
"show": true,
"total": false,
"values": false
},
"lines": true,
"linewidth": 1,
"links": [],
"nullPointMode": "null",
"percentage": false,
"pointradius": 5,
"points": false,
"renderer": "flot",
"seriesOverrides": [
{
"alias": "/.*_QPS/",
"yaxis": 2
}
],
"spaceLength": 10,
"stack": false,
"steppedLine": false,
"targets": [
{
"expr": "sum([[app]]_metanode_OpMetaCreateDentry{cluster=~\"$cluster\"})",
"format": "time_series",
"intervalFactor": 1,
"legendFormat": "CreateDentryCreateDentry",
"refId": "A"
},
{
"expr": "sum([[app]]_metanode_OpMetaDeleteDentry{cluster=~\"$cluster\"})",
"format": "time_series",
"intervalFactor": 1,
"legendFormat": "DeleteDentry",
"refId": "B"
},
{
"expr": "sum([[app]]_metanode_OpMetaUpdateDentry{cluster=~\"$cluster\"})",
"format": "time_series",
"intervalFactor": 1,
"legendFormat": "UpdateDentry",
"refId": "C"
},
{
"expr": "sum(rate([[app]]_metanode_OpMetaCreateDentry{cluster=~\"$cluster\"}[1m]))",
"format": "time_series",
"intervalFactor": 1,
"legendFormat": "CreateDentry_QPS",
"refId": "D"
},
{
"expr": "sum(rate([[app]]_metanode_OpMetaDeleteDentry{cluster=~\"$cluster\"}[1m]))",
"format": "time_series",
"intervalFactor": 1,
"legendFormat": "DeleteDentry_QPS",
"refId": "E"
},
{
"expr": "sum(rate([[app]]_metanode_OpMetaUpdateDentry{cluster=~\"$cluster\"}[1m]))",
"format": "time_series",
"intervalFactor": 1,
"legendFormat": "UpdateDentry_QPS",
"refId": "F"
}
],
"thresholds": [],
"timeFrom": null,
"timeShift": null,
"title": "metanode_OpMetaDentry",
"tooltip": {
"shared": true,
"sort": 0,
"value_type": "individual"
},
"type": "graph",
"xaxis": {
"buckets": null,
"mode": "time",
"name": null,
"show": true,
"values": []
},
"yaxes": [
{
"format": "ns",
"label": null,
"logBase": 1,
"max": null,
"min": null,
"show": true
},
{
"format": "ops",
"label": null,
"logBase": 1,
"max": null,
"min": null,
"show": true
}
],
"yaxis": {
"align": false,
"alignLevel": null
}
},
{
"aliasColors": {},
"bars": false,
"dashLength": 10,
"dashes": false,
"datasource": "default",
"fill": 1,
"gridPos": {
"h": 6,
"w": 7,
"x": 14,
"y": 23
},
"id": 79,
"legend": {
"avg": false,
"current": false,
"max": false,
"min": false,
"show": true,
"total": false,
"values": false
},
"lines": true,
"linewidth": 1,
"links": [],
"nullPointMode": "null",
"percentage": false,
"pointradius": 5,
"points": false,
"renderer": "flot",
"seriesOverrides": [
{
"alias": "/.*_QPS/",
"yaxis": 2
}
],
"spaceLength": 10,
"stack": false,
"steppedLine": false,
"targets": [
{
"expr": "sum([[app]]_dataNode_OpCreateExtent{cluster=~\"$cluster\"})",
"format": "time_series",
"intervalFactor": 1,
"legendFormat": "CreateExtent",
"refId": "A"
},
{
"expr": "sum([[app]]_dataNode_OpMarkDelete{cluster=~\"$cluster\"})",
"format": "time_series",
"intervalFactor": 1,
"legendFormat": "MarkDeleteExtent",
"refId": "B"
},
{
"expr": "sum([[app]]_dataNode_OpRandomWrite{cluster=~\"$cluster\"})",
"format": "time_series",
"intervalFactor": 1,
"legendFormat": "RandomWrite",
"refId": "D"
},
{
"expr": "sum([[app]]_dataNode_OpWrite{cluster=~\"$cluster\"})",
"format": "time_series",
"intervalFactor": 1,
"legendFormat": "AppendWrite",
"refId": "C"
},
{
"expr": "sum([[app]]_dataNode_OpStreamRead{cluster=~\"$cluster\"})",
"format": "time_series",
"intervalFactor": 1,
"legendFormat": "Read",
"refId": "F"
},
{
"expr": "sum(rate([[app]]_datanode_OpCreateExtent{cluster=~\"$cluster\"}[1m]))",
"format": "time_series",
"intervalFactor": 1,
"legendFormat": "CreateExtent_QPS",
"refId": "E"
},
{
"expr": "sum(rate([[app]]_dataNode_OpMarkDelete{cluster=~\"$cluster\"}[1m]))",
"format": "time_series",
"intervalFactor": 1,
"legendFormat": "MarkDeleteExtent_QPS",
"refId": "G"
},
{
"expr": "sum(rate([[app]]_dataNode_OpRandomWrite{cluster=~\"$cluster\"}[1m]))",
"format": "time_series",
"intervalFactor": 1,
"legendFormat": "RandomWrite_QPS",
"refId": "H"
},
{
"expr": "sum(rate([[app]]_dataNode_OpWrite{cluster=~\"$cluster\"}[1m]))",
"format": "time_series",
"intervalFactor": 1,
"legendFormat": "AppendWrite_QPS",
"refId": "I"
},
{
"expr": "sum(rate([[app]]_dataNode_OpStreamRead{cluster=~\"$cluster\"}[1m]))",
"format": "time_series",
"intervalFactor": 1,
"legendFormat": "Read_QPS",
"refId": "J"
}
],
"thresholds": [],
"timeFrom": null,
"timeShift": null,
"title": "datanode_OpExtent",
"tooltip": {
"shared": true,
"sort": 0,
"value_type": "individual"
},
"type": "graph",
"xaxis": {
"buckets": null,
"mode": "time",
"name": null,
"show": true,
"values": []
},
"yaxes": [
{
"format": "ns",
"label": null,
"logBase": 1,
"max": null,
"min": null,
"show": true
},
{
"format": "ops",
"label": null,
"logBase": 1,
"max": null,
"min": null,
"show": true
}
],
"yaxis": {
"align": false,
"alignLevel": null
}
},
{
"aliasColors": {},
"bars": false,
"dashLength": 10,
"dashes": false,
"datasource": "default",
"fill": 1,
"gridPos": {
"h": 6,
"w": 7,
"x": 0,
"y": 29
},
"id": 70,
"legend": {
"avg": false,
"current": false,
"max": false,
"min": false,
"show": true,
"total": false,
"values": false
},
"lines": true,
"linewidth": 1,
"links": [],
"nullPointMode": "null",
"percentage": false,
"pointradius": 5,
"points": false,
"renderer": "flot",
"seriesOverrides": [],
"spaceLength": 10,
"stack": false,
"steppedLine": false,
"targets": [
{
"expr": "sum([[app]]_metanode_OpCreateMetaPartition{cluster=~\"$cluster\", instance=\"$instance\"})",
"format": "time_series",
"intervalFactor": 1,
"legendFormat": "Create",
"refId": "A"
},
{
"expr": "sum([[app]]_metanode_OpLoadMetaPartition{cluster=~\"$cluster\", instance=\"$instance\"})",
"format": "time_series",
"intervalFactor": 1,
"legendFormat": "Load",
"refId": "B"
}
],
"thresholds": [],
"timeFrom": null,
"timeShift": null,
"title": "metanode_OpMetaPartition",
"tooltip": {
"shared": true,
"sort": 0,
"value_type": "individual"
},
"type": "graph",
"xaxis": {
"buckets": null,
"mode": "time",
"name": null,
"show": true,
"values": []
},
"yaxes": [
{
"format": "ns",
"label": null,
"logBase": 1,
"max": null,
"min": null,
"show": true
},
{
"format": "short",
"label": null,
"logBase": 1,
"max": null,
"min": null,
"show": true
}
],
"yaxis": {
"align": false,
"alignLevel": null
}
},
{
"aliasColors": {},
"bars": false,
"dashLength": 10,
"dashes": false,
"datasource": "default",
"fill": 1,
"gridPos": {
"h": 6,
"w": 7,
"x": 7,
"y": 29
},
"id": 75,
"legend": {
"avg": false,
"current": false,
"max": false,
"min": false,
"show": true,
"total": false,
"values": false
},
"lines": true,
"linewidth": 1,
"links": [],
"nullPointMode": "null",
"percentage": false,
"pointradius": 5,
"points": false,
"renderer": "flot",
"seriesOverrides": [],
"spaceLength": 10,
"stack": false,
"steppedLine": false,
"targets": [
{
"expr": "sum([[app]]_dataNode_OpLoadDataPartition{cluster=~\"$cluster\"})",
"format": "time_series",
"intervalFactor": 1,
"legendFormat": "LoadDataPartition",
"refId": "G"
},
{
"expr": "sum([[app]]_dataNode_OpDataNodeHeartbeat{cluster=~\"$cluster\"})",
"format": "time_series",
"intervalFactor": 1,
"legendFormat": "DataNodeHeartbeat",
"refId": "F"
},
{
"expr": "sum([[app]]_dataNode_OpCreateDataPartition{cluster=~\"$cluster\"})",
"format": "time_series",
"intervalFactor": 1,
"legendFormat": "CreateDataPartition",
"refId": "A"
},
{
"expr": "sum([[app]]_dataNode_OpAddDataPartitionRaftMember{cluster=~\"$cluster\"})",
"format": "time_series",
"intervalFactor": 1,
"legendFormat": "AddDataPartitionRaftMember",
"refId": "B"
},
{
"expr": "sum([[app]]_dataNode_OpDeleteDataPartition{cluster=~\"$cluster\"})",
"format": "time_series",
"intervalFactor": 1,
"legendFormat": "DeleteDataPartition",
"refId": "C"
}
],
"thresholds": [],
"timeFrom": null,
"timeShift": null,
"title": "datanode_OpDataPartition",
"tooltip": {
"shared": true,
"sort": 0,
"value_type": "individual"
},
"type": "graph",
"xaxis": {
"buckets": null,
"mode": "time",
"name": null,
"show": true,
"values": []
},
"yaxes": [
{
"format": "ns",
"label": null,
"logBase": 1,
"max": null,
"min": null,
"show": true
},
{
"format": "short",
"label": null,
"logBase": 1,
"max": null,
"min": null,
"show": true
}
],
"yaxis": {
"align": false,
"alignLevel": null
}
},
{
"collapsed": true,
"gridPos": {
"h": 1,
"w": 24,
"x": 0,
"y": 35
},
"id": 34,
"panels": [
{
"aliasColors": {},
"bars": true,
"dashLength": 10,
"dashes": false,
"datasource": null,
"fill": 1,
"gridPos": {
"h": 7,
"w": 7,
"x": 0,
"y": 3
},
"id": 162,
"legend": {
"avg": false,
"current": false,
"max": false,
"min": false,
"show": true,
"total": false,
"values": false
},
"lines": false,
"linewidth": 1,
"links": [],
"nullPointMode": "null",
"percentage": false,
"pointradius": 5,
"points": false,
"renderer": "flot",
"seriesOverrides": [
{
"alias": "master_count",
"yaxis": 2
}
],
"spaceLength": 10,
"stack": false,
"steppedLine": false,
"targets": [
{
"expr": "go_info{cluster=\"$cluster\", role=\"master\"}",
"format": "time_series",
"hide": false,
"intervalFactor": 1,
"legendFormat": "master_{{instance}}",
"refId": "C"
},
{
"expr": "count(go_info{cluster=~\"$cluster\", role=~\"master\"})",
"format": "time_series",
"hide": false,
"intervalFactor": 1,
"legendFormat": "master_count",
"refId": "G"
}
],
"thresholds": [],
"timeFrom": null,
"timeShift": null,
"title": "[[cluster]]_[[role]]_nodes",
"tooltip": {
"shared": true,
"sort": 0,
"value_type": "individual"
},
"type": "graph",
"xaxis": {
"buckets": null,
"mode": "time",
"name": null,
"show": true,
"values": []
},
"yaxes": [
{
"format": "locale",
"label": null,
"logBase": 1,
"max": null,
"min": null,
"show": true
},
{
"format": "locale",
"label": null,
"logBase": 1,
"max": null,
"min": null,
"show": true
}
],
"yaxis": {
"align": false,
"alignLevel": null
}
},
{
"aliasColors": {},
"bars": true,
"dashLength": 10,
"dashes": false,
"datasource": null,
"fill": 1,
"gridPos": {
"h": 7,
"w": 7,
"x": 7,
"y": 3
},
"id": 163,
"legend": {
"avg": false,
"current": false,
"max": false,
"min": false,
"show": true,
"total": false,
"values": false
},
"lines": false,
"linewidth": 1,
"links": [],
"nullPointMode": "null",
"percentage": false,
"pointradius": 5,
"points": false,
"renderer": "flot",
"seriesOverrides": [],
"spaceLength": 10,
"stack": true,
"steppedLine": false,
"targets": [
{
"expr": "go_info{cluster=\"$cluster\", role=\"metanode\"}",
"format": "time_series",
"intervalFactor": 1,
"legendFormat": "meta_{{instance}}",
"refId": "C"
},
{
"expr": "sum([[app]]_master_metaNodes_count{cluster=~\"$cluster\"})",
"format": "time_series",
"hide": true,
"intervalFactor": 1,
"legendFormat": "metaNode",
"refId": "B"
},
{
"expr": "sum([[app]]_master_dataNodes_count{cluster=~\"$cluster\"})",
"format": "time_series",
"hide": true,
"intervalFactor": 1,
"legendFormat": "datanode",
"refId": "A"
},
{
"expr": "[[app]]_master_metaNodes_count",
"format": "time_series",
"hide": true,
"intervalFactor": 1,
"legendFormat": "meta_count",
"refId": "D"
},
{
"expr": "[[app]]_master_dataNodes_count",
"format": "time_series",
"hide": true,
"intervalFactor": 1,
"legendFormat": "datanode_count",
"refId": "E"
},
{
"expr": "count(process_start_time_seconds{role=\"fuseclient\"})",
"format": "time_series",
"hide": true,
"intervalFactor": 1,
"legendFormat": "client_count",
"refId": "F"
}
],
"thresholds": [],
"timeFrom": null,
"timeShift": null,
"title": "[[cluster]]_metanode_alive",
"tooltip": {
"shared": true,
"sort": 0,
"value_type": "individual"
},
"type": "graph",
"xaxis": {
"buckets": null,
"mode": "time",
"name": null,
"show": true,
"values": []
},
"yaxes": [
{
"format": "locale",
"label": null,
"logBase": 1,
"max": null,
"min": null,
"show": true
},
{
"format": "short",
"label": null,
"logBase": 1,
"max": null,
"min": null,
"show": true
}
],
"yaxis": {
"align": false,
"alignLevel": null
}
},
{
"aliasColors": {},
"bars": true,
"dashLength": 10,
"dashes": false,
"datasource": null,
"fill": 1,
"gridPos": {
"h": 7,
"w": 7,
"x": 0,
"y": 10
},
"id": 164,
"legend": {
"avg": false,
"current": false,
"max": false,
"min": false,
"show": true,
"total": false,
"values": false
},
"lines": false,
"linewidth": 1,
"links": [],
"nullPointMode": "null",
"percentage": false,
"pointradius": 5,
"points": false,
"renderer": "flot",
"seriesOverrides": [],
"spaceLength": 10,
"stack": true,
"steppedLine": false,
"targets": [
{
"expr": "go_info{cluster=\"$cluster\", role=\"dataNode\"}",
"format": "time_series",
"intervalFactor": 1,
"legendFormat": "{{instance}}",
"refId": "C"
},
{
"expr": "sum([[app]]_master_metaNodes_count{cluster=~\"$cluster\"})",
"format": "time_series",
"hide": true,
"intervalFactor": 1,
"legendFormat": "metaNode",
"refId": "B"
},
{
"expr": "sum([[app]]_master_dataNodes_count{cluster=~\"$cluster\"})",
"format": "time_series",
"hide": true,
"intervalFactor": 1,
"legendFormat": "datanode",
"refId": "A"
},
{
"expr": "[[app]]_master_metaNodes_count",
"format": "time_series",
"hide": true,
"intervalFactor": 1,
"legendFormat": "meta_count",
"refId": "D"
},
{
"expr": "[[app]]_master_dataNodes_count",
"format": "time_series",
"hide": true,
"intervalFactor": 1,
"legendFormat": "datanode_count",
"refId": "E"
},
{
"expr": "count(process_start_time_seconds{role=\"fuseclient\"})",
"format": "time_series",
"hide": true,
"intervalFactor": 1,
"legendFormat": "client_count",
"refId": "F"
}
],
"thresholds": [],
"timeFrom": null,
"timeShift": null,
"title": "[[cluster]]_datanode_alive",
"tooltip": {
"shared": true,
"sort": 0,
"value_type": "individual"
},
"type": "graph",
"xaxis": {
"buckets": null,
"mode": "time",
"name": null,
"show": true,
"values": []
},
"yaxes": [
{
"format": "locale",
"label": null,
"logBase": 1,
"max": null,
"min": null,
"show": true
},
{
"format": "short",
"label": null,
"logBase": 1,
"max": null,
"min": null,
"show": true
}
],
"yaxis": {
"align": false,
"alignLevel": null
}
},
{
"aliasColors": {},
"bars": true,
"dashLength": 10,
"dashes": false,
"datasource": null,
"fill": 1,
"gridPos": {
"h": 7,
"w": 7,
"x": 7,
"y": 10
},
"id": 165,
"legend": {
"avg": false,
"current": false,
"max": false,
"min": false,
"show": true,
"total": false,
"values": false
},
"lines": false,
"linewidth": 1,
"links": [],
"nullPointMode": "null",
"percentage": false,
"pointradius": 5,
"points": false,
"renderer": "flot",
"seriesOverrides": [],
"spaceLength": 10,
"stack": true,
"steppedLine": false,
"targets": [
{
"expr": "go_info{cluster=\"$cluster\", role=\"fuseclient\"}",
"format": "time_series",
"intervalFactor": 1,
"legendFormat": "fuseclient_{{instance}}",
"refId": "C"
},
{
"expr": "sum([[app]]_master_metaNodes_count{cluster=~\"$cluster\"})",
"format": "time_series",
"hide": true,
"intervalFactor": 1,
"legendFormat": "metaNode",
"refId": "B"
},
{
"expr": "sum([[app]]_master_dataNodes_count{cluster=~\"$cluster\"})",
"format": "time_series",
"hide": true,
"intervalFactor": 1,
"legendFormat": "datanode",
"refId": "A"
},
{
"expr": "[[app]]_master_metaNodes_count",
"format": "time_series",
"hide": true,
"intervalFactor": 1,
"legendFormat": "meta_count",
"refId": "D"
},
{
"expr": "[[app]]_master_dataNodes_count",
"format": "time_series",
"hide": true,
"intervalFactor": 1,
"legendFormat": "datanode_count",
"refId": "E"
},
{
"expr": "count(process_start_time_seconds{role=\"fuseclient\"})",
"format": "time_series",
"hide": true,
"intervalFactor": 1,
"legendFormat": "client_count",
"refId": "F"
}
],
"thresholds": [],
"timeFrom": null,
"timeShift": null,
"title": "[[cluster]]_fuseclient_alive",
"tooltip": {
"shared": true,
"sort": 0,
"value_type": "individual"
},
"type": "graph",
"xaxis": {
"buckets": null,
"mode": "time",
"name": null,
"show": true,
"values": []
},
"yaxes": [
{
"format": "locale",
"label": null,
"logBase": 1,
"max": null,
"min": null,
"show": true
},
{
"format": "short",
"label": null,
"logBase": 1,
"max": null,
"min": null,
"show": true
}
],
"yaxis": {
"align": false,
"alignLevel": null
}
}
],
"title": "Master",
"type": "row"
},
{
"collapsed": true,
"gridPos": {
"h": 1,
"w": 24,
"x": 0,
"y": 36
},
"id": 36,
"panels": [
{
"aliasColors": {},
"bars": false,
"dashLength": 10,
"dashes": false,
"datasource": null,
"fill": 1,
"gridPos": {
"h": 8,
"w": 8,
"x": 0,
"y": 4
},
"id": 117,
"legend": {
"avg": false,
"current": false,
"max": false,
"min": false,
"show": true,
"total": false,
"values": false
},
"lines": true,
"linewidth": 1,
"links": [],
"nullPointMode": "null",
"percentage": false,
"pointradius": 5,
"points": false,
"renderer": "flot",
"seriesOverrides": [
{
"alias": "/.*_ops_.*/",
"yaxis": 2
}
],
"spaceLength": 10,
"stack": false,
"steppedLine": false,
"targets": [
{
"expr": "[[app]]_metanode_OpMetaCreateInode{app=~\"$app\", cluster=\"$cluster\"}",
"format": "time_series",
"hide": false,
"intervalFactor": 1,
"legendFormat": "OpMetaCreateInode_{{instance}}",
"refId": "B"
},
{
"expr": "rate([[app]]_metanode_OpMetaCreateInode_count{app=~\"$app\", cluster=\"$cluster\"}[1m])",
"format": "time_series",
"hide": false,
"intervalFactor": 1,
"legendFormat": "OpMetaCreateInode_ops_{{instance}}",
"refId": "A"
}
],
"thresholds": [],
"timeFrom": null,
"timeShift": null,
"title": "OpMetaCreateInode",
"tooltip": {
"shared": true,
"sort": 0,
"value_type": "individual"
},
"type": "graph",
"xaxis": {
"buckets": null,
"mode": "time",
"name": null,
"show": true,
"values": []
},
"yaxes": [
{
"format": "ns",
"label": null,
"logBase": 1,
"max": null,
"min": null,
"show": true
},
{
"format": "ops",
"label": null,
"logBase": 1,
"max": null,
"min": null,
"show": true
}
],
"yaxis": {
"align": false,
"alignLevel": null
}
},
{
"aliasColors": {},
"bars": false,
"dashLength": 10,
"dashes": false,
"datasource": null,
"fill": 1,
"gridPos": {
"h": 8,
"w": 8,
"x": 8,
"y": 4
},
"id": 44,
"legend": {
"avg": false,
"current": false,
"max": false,
"min": false,
"show": true,
"total": false,
"values": false
},
"lines": true,
"linewidth": 1,
"links": [],
"nullPointMode": "null",
"percentage": false,
"pointradius": 5,
"points": false,
"renderer": "flot",
"seriesOverrides": [],
"spaceLength": 10,
"stack": false,
"steppedLine": false,
"targets": [
{
"expr": "[[app]]_metanode_OpMetaBatchInodeGet{app=~\"$app\", cluster=\"$cluster\"}",
"format": "time_series",
"intervalFactor": 1,
"legendFormat": "OpMetaBatchInodeGet_{{instance}}",
"refId": "A"
},
{
"expr": "[[app]]_metanode_OpMetaCreateInode{app=~\"$app\", cluster=\"$cluster\"}",
"format": "time_series",
"hide": true,
"intervalFactor": 1,
"legendFormat": "OpMetaCreateInode",
"refId": "B"
},
{
"expr": "[[app]]_metanode_OpMetaDeleteInode{app=~\"$app\", cluster=\"$cluster\"}",
"format": "time_series",
"hide": true,
"intervalFactor": 1,
"legendFormat": "OpMetaDeleteInode_{{instance}}",
"refId": "C"
},
{
"expr": "[[app]]_metanode_OpMetaEvictInode{app=~\"$app\", cluster=\"$cluster\"}",
"format": "time_series",
"hide": true,
"intervalFactor": 1,
"legendFormat": "OpMetaEvictInode_{{instance}}",
"refId": "D"
},
{
"expr": "[[app]]_metanode_OpMetaInodeGet{app=~\"$app\", cluster=\"$cluster\"}",
"format": "time_series",
"hide": true,
"intervalFactor": 1,
"legendFormat": "OpMetaInodeGet_{{instance}}",
"refId": "E"
},
{
"expr": "[[app]]_metanode_OpMetaLinkInode{app=~\"$app\", cluster=\"$cluster\"}",
"format": "time_series",
"hide": true,
"intervalFactor": 1,
"legendFormat": "OpMetaLinkInode_{{instance}}",
"refId": "F"
}
],
"thresholds": [],
"timeFrom": null,
"timeShift": null,
"title": "OpMetaBatchInodeGet",
"tooltip": {
"shared": true,
"sort": 0,
"value_type": "individual"
},
"type": "graph",
"xaxis": {
"buckets": null,
"mode": "time",
"name": null,
"show": true,
"values": []
},
"yaxes": [
{
"format": "ns",
"label": null,
"logBase": 1,
"max": null,
"min": null,
"show": true
},
{
"format": "short",
"label": null,
"logBase": 1,
"max": null,
"min": null,
"show": true
}
],
"yaxis": {
"align": false,
"alignLevel": null
}
},
{
"aliasColors": {},
"bars": false,
"dashLength": 10,
"dashes": false,
"datasource": null,
"fill": 1,
"gridPos": {
"h": 8,
"w": 8,
"x": 16,
"y": 4
},
"id": 119,
"legend": {
"avg": false,
"current": false,
"max": false,
"min": false,
"show": true,
"total": false,
"values": false
},
"lines": true,
"linewidth": 1,
"links": [],
"nullPointMode": "null",
"percentage": false,
"pointradius": 5,
"points": false,
"renderer": "flot",
"seriesOverrides": [
{
"alias": "/.*_ops_.*/",
"yaxis": 2
}
],
"spaceLength": 10,
"stack": false,
"steppedLine": false,
"targets": [
{
"expr": "[[app]]_metanode_OpMetaBatchInodeGet{app=~\"$app\", cluster=\"$cluster\"}",
"format": "time_series",
"hide": true,
"intervalFactor": 1,
"legendFormat": "OpMetaBatchInodeGet_{{instance}}",
"refId": "A"
},
{
"expr": "[[app]]_metanode_OpMetaCreateInode{app=~\"$app\", cluster=\"$cluster\"}",
"format": "time_series",
"hide": true,
"intervalFactor": 1,
"legendFormat": "OpMetaCreateInode",
"refId": "B"
},
{
"expr": "[[app]]_metanode_OpMetaDeleteInode{app=~\"$app\", cluster=\"$cluster\"}",
"format": "time_series",
"hide": true,
"intervalFactor": 1,
"legendFormat": "OpMetaDeleteInode_{{instance}}",
"refId": "C"
},
{
"expr": "[[app]]_metanode_OpMetaEvictInode{app=~\"$app\", cluster=\"$cluster\"}",
"format": "time_series",
"hide": true,
"intervalFactor": 1,
"legendFormat": "OpMetaEvictInode_{{instance}}",
"refId": "D"
},
{
"expr": "[[app]]_metanode_OpMetaInodeGet{app=~\"$app\", cluster=\"$cluster\"}",
"format": "time_series",
"hide": true,
"intervalFactor": 1,
"legendFormat": "OpMetaInodeGet_{{instance}}",
"refId": "E"
},
{
"expr": "[[app]]_metanode_OpMetaLinkInode{app=~\"$app\", cluster=\"$cluster\"}",
"format": "time_series",
"hide": false,
"intervalFactor": 1,
"legendFormat": "OpMetaLinkInode_{{instance}}",
"refId": "F"
},
{
"expr": "rate([[app]]_metanode_OpMetaLinkInode_count{app=~\"$app\", cluster=\"$cluster\"}[1m])",
"format": "time_series",
"hide": false,
"intervalFactor": 1,
"legendFormat": "OpMetaLinkInode_ops_{{instance}}",
"refId": "G"
}
],
"thresholds": [],
"timeFrom": null,
"timeShift": null,
"title": "OpMetaLinkInode",
"tooltip": {
"shared": true,
"sort": 0,
"value_type": "individual"
},
"type": "graph",
"xaxis": {
"buckets": null,
"mode": "time",
"name": null,
"show": true,
"values": []
},
"yaxes": [
{
"format": "ns",
"label": null,
"logBase": 1,
"max": null,
"min": null,
"show": true
},
{
"format": "ops",
"label": null,
"logBase": 1,
"max": null,
"min": null,
"show": true
}
],
"yaxis": {
"align": false,
"alignLevel": null
}
},
{
"aliasColors": {},
"bars": false,
"dashLength": 10,
"dashes": false,
"datasource": null,
"fill": 1,
"gridPos": {
"h": 8,
"w": 8,
"x": 0,
"y": 12
},
"id": 120,
"legend": {
"avg": false,
"current": false,
"max": false,
"min": false,
"show": true,
"total": false,
"values": false
},
"lines": true,
"linewidth": 1,
"links": [],
"nullPointMode": "null",
"percentage": false,
"pointradius": 5,
"points": false,
"renderer": "flot",
"seriesOverrides": [],
"spaceLength": 10,
"stack": false,
"steppedLine": false,
"targets": [
{
"expr": "[[app]]_metanode_OpMetaBatchInodeGet{app=~\"$app\", cluster=\"$cluster\"}",
"format": "time_series",
"hide": true,
"intervalFactor": 1,
"legendFormat": "OpMetaBatchInodeGet_{{instance}}",
"refId": "A"
},
{
"expr": "[[app]]_metanode_OpMetaCreateInode{app=~\"$app\", cluster=\"$cluster\"}",
"format": "time_series",
"hide": true,
"intervalFactor": 1,
"legendFormat": "OpMetaCreateInode",
"refId": "B"
},
{
"expr": "[[app]]_metanode_OpMetaDeleteInode{app=~\"$app\", cluster=\"$cluster\"}",
"format": "time_series",
"hide": true,
"intervalFactor": 1,
"legendFormat": "OpMetaDeleteInode_{{instance}}",
"refId": "C"
},
{
"expr": "[[app]]_metanode_OpMetaEvictInode{app=~\"$app\", cluster=\"$cluster\"}",
"format": "time_series",
"hide": false,
"intervalFactor": 1,
"legendFormat": "OpMetaEvictInode_{{instance}}",
"refId": "D"
},
{
"expr": "[[app]]_metanode_OpMetaInodeGet{app=~\"$app\", cluster=\"$cluster\"}",
"format": "time_series",
"hide": true,
"intervalFactor": 1,
"legendFormat": "OpMetaInodeGet_{{instance}}",
"refId": "E"
},
{
"expr": "[[app]]_metanode_OpMetaLinkInode{app=~\"$app\", cluster=\"$cluster\"}",
"format": "time_series",
"hide": true,
"intervalFactor": 1,
"legendFormat": "OpMetaLinkInode_{{instance}}",
"refId": "F"
}
],
"thresholds": [],
"timeFrom": null,
"timeShift": null,
"title": "OpMetaEvictInode",
"tooltip": {
"shared": true,
"sort": 0,
"value_type": "individual"
},
"type": "graph",
"xaxis": {
"buckets": null,
"mode": "time",
"name": null,
"show": true,
"values": []
},
"yaxes": [
{
"format": "ns",
"label": null,
"logBase": 1,
"max": null,
"min": null,
"show": true
},
{
"format": "short",
"label": null,
"logBase": 1,
"max": null,
"min": null,
"show": true
}
],
"yaxis": {
"align": false,
"alignLevel": null
}
},
{
"aliasColors": {},
"bars": false,
"dashLength": 10,
"dashes": false,
"datasource": null,
"fill": 1,
"gridPos": {
"h": 8,
"w": 8,
"x": 8,
"y": 12
},
"id": 121,
"legend": {
"avg": false,
"current": false,
"max": false,
"min": false,
"show": true,
"total": false,
"values": false
},
"lines": true,
"linewidth": 1,
"links": [],
"nullPointMode": "null",
"percentage": false,
"pointradius": 5,
"points": false,
"renderer": "flot",
"seriesOverrides": [],
"spaceLength": 10,
"stack": false,
"steppedLine": false,
"targets": [
{
"expr": "[[app]]_metanode_OpCreateMetaPartition{app=\"$app\", cluster=\"$cluster\"}",
"format": "time_series",
"hide": true,
"intervalFactor": 1,
"legendFormat": "OpCreateMetaPartition_{{instance}}",
"refId": "A"
},
{
"expr": "[[app]]_metanode_OpLoadMetaPartition{app=\"$app\", cluster=\"$cluster\"}",
"format": "time_series",
"hide": false,
"intervalFactor": 1,
"legendFormat": "OpLoadMetaPartition_{{instance}}",
"refId": "B"
}
],
"thresholds": [],
"timeFrom": null,
"timeShift": null,
"title": "OpLoadMetaPartition",
"tooltip": {
"shared": true,
"sort": 0,
"value_type": "individual"
},
"type": "graph",
"xaxis": {
"buckets": null,
"mode": "time",
"name": null,
"show": true,
"values": []
},
"yaxes": [
{
"format": "ns",
"label": null,
"logBase": 1,
"max": null,
"min": null,
"show": true
},
{
"format": "short",
"label": null,
"logBase": 1,
"max": null,
"min": null,
"show": true
}
],
"yaxis": {
"align": false,
"alignLevel": null
}
},
{
"aliasColors": {},
"bars": false,
"dashLength": 10,
"dashes": false,
"datasource": null,
"fill": 1,
"gridPos": {
"h": 8,
"w": 8,
"x": 16,
"y": 12
},
"id": 58,
"legend": {
"avg": false,
"current": false,
"max": false,
"min": false,
"show": true,
"total": false,
"values": false
},
"lines": true,
"linewidth": 1,
"links": [],
"nullPointMode": "null",
"percentage": false,
"pointradius": 5,
"points": false,
"renderer": "flot",
"seriesOverrides": [],
"spaceLength": 10,
"stack": false,
"steppedLine": false,
"targets": [
{
"expr": "[[app]]_metanode_OpCreateMetaPartition{app=\"$app\", cluster=\"$cluster\"}",
"format": "time_series",
"intervalFactor": 1,
"legendFormat": "OpCreateMetaPartition_{{instance}}",
"refId": "A"
},
{
"expr": "[[app]]_metanode_OpLoadMetaPartition{app=\"$app\", cluster=\"$cluster\"}",
"format": "time_series",
"hide": true,
"intervalFactor": 1,
"legendFormat": "OpLoadMetaPartition_{{instance}}",
"refId": "B"
}
],
"thresholds": [],
"timeFrom": null,
"timeShift": null,
"title": "OpCreateMetaPartition",
"tooltip": {
"shared": true,
"sort": 0,
"value_type": "individual"
},
"type": "graph",
"xaxis": {
"buckets": null,
"mode": "time",
"name": null,
"show": true,
"values": []
},
"yaxes": [
{
"format": "ns",
"label": null,
"logBase": 1,
"max": null,
"min": null,
"show": true
},
{
"format": "short",
"label": null,
"logBase": 1,
"max": null,
"min": null,
"show": true
}
],
"yaxis": {
"align": false,
"alignLevel": null
}
},
{
"aliasColors": {},
"bars": false,
"dashLength": 10,
"dashes": false,
"datasource": null,
"fill": 1,
"gridPos": {
"h": 8,
"w": 8,
"x": 0,
"y": 20
},
"id": 88,
"legend": {
"avg": false,
"current": false,
"max": false,
"min": false,
"show": true,
"total": false,
"values": false
},
"lines": true,
"linewidth": 1,
"links": [],
"nullPointMode": "null",
"percentage": false,
"pointradius": 5,
"points": false,
"renderer": "flot",
"seriesOverrides": [],
"spaceLength": 10,
"stack": false,
"steppedLine": false,
"targets": [
{
"expr": "[[app]]_metanode_OpMetaExtentsList{app=~\"$app\", cluster=\"$cluster\", instance=\"$instance\"}",
"format": "time_series",
"intervalFactor": 1,
"legendFormat": "OpMetaExtentsList_{{instance}}",
"refId": "B"
}
],
"thresholds": [],
"timeFrom": null,
"timeShift": null,
"title": "OpMetaExtentsList",
"tooltip": {
"shared": true,
"sort": 0,
"value_type": "individual"
},
"type": "graph",
"xaxis": {
"buckets": null,
"mode": "time",
"name": null,
"show": true,
"values": []
},
"yaxes": [
{
"format": "ns",
"label": null,
"logBase": 1,
"max": null,
"min": null,
"show": true
},
{
"format": "short",
"label": null,
"logBase": 1,
"max": null,
"min": null,
"show": true
}
],
"yaxis": {
"align": false,
"alignLevel": null
}
},
{
"aliasColors": {},
"bars": false,
"dashLength": 10,
"dashes": false,
"datasource": null,
"fill": 1,
"gridPos": {
"h": 8,
"w": 8,
"x": 8,
"y": 20
},
"id": 50,
"legend": {
"avg": false,
"current": false,
"max": false,
"min": false,
"show": true,
"total": false,
"values": false
},
"lines": true,
"linewidth": 1,
"links": [],
"nullPointMode": "null",
"percentage": false,
"pointradius": 5,
"points": false,
"renderer": "flot",
"seriesOverrides": [],
"spaceLength": 10,
"stack": false,
"steppedLine": false,
"targets": [
{
"expr": "[[app]]_metanode_OpMetaExtentsAdd{app=~\"$app\", cluster=\"$cluster\"}",
"format": "time_series",
"intervalFactor": 1,
"legendFormat": "OpMetaExtentsAdd_{{instance}}",
"refId": "A"
},
{
"expr": "[[app]]_metanode_OpMetaExtentsList{app=~\"$app\", cluster=\"$cluster\"}",
"format": "time_series",
"hide": true,
"intervalFactor": 1,
"legendFormat": "OpMetaExtentsList_{{instance}}",
"refId": "B"
}
],
"thresholds": [],
"timeFrom": null,
"timeShift": null,
"title": "OpMetaExtentsAdd",
"tooltip": {
"shared": true,
"sort": 0,
"value_type": "individual"
},
"type": "graph",
"xaxis": {
"buckets": null,
"mode": "time",
"name": null,
"show": true,
"values": []
},
"yaxes": [
{
"format": "ns",
"label": null,
"logBase": 1,
"max": null,
"min": null,
"show": true
},
{
"format": "short",
"label": null,
"logBase": 1,
"max": null,
"min": null,
"show": true
}
],
"yaxis": {
"align": false,
"alignLevel": null
}
},
{
"aliasColors": {},
"bars": false,
"dashLength": 10,
"dashes": false,
"datasource": null,
"fill": 1,
"gridPos": {
"h": 8,
"w": 8,
"x": 16,
"y": 20
},
"id": 126,
"legend": {
"avg": false,
"current": false,
"max": false,
"min": false,
"show": true,
"total": false,
"values": false
},
"lines": true,
"linewidth": 1,
"links": [],
"nullPointMode": "null",
"percentage": false,
"pointradius": 5,
"points": false,
"renderer": "flot",
"seriesOverrides": [],
"spaceLength": 10,
"stack": false,
"steppedLine": false,
"targets": [
{
"expr": "[[app]]_metanode_OpMetaNodeHeartbeat{app=~\"$app\", cluster=\"$cluster\"}",
"format": "time_series",
"hide": false,
"intervalFactor": 1,
"legendFormat": "OpMetaNodeHeartbeat_{{instance}}",
"refId": "C"
}
],
"thresholds": [],
"timeFrom": null,
"timeShift": null,
"title": "OpMetaNodeHeartbeat",
"tooltip": {
"shared": true,
"sort": 0,
"value_type": "individual"
},
"type": "graph",
"xaxis": {
"buckets": null,
"mode": "time",
"name": null,
"show": true,
"values": []
},
"yaxes": [
{
"format": "ns",
"label": null,
"logBase": 1,
"max": null,
"min": null,
"show": true
},
{
"format": "short",
"label": null,
"logBase": 1,
"max": null,
"min": null,
"show": true
}
],
"yaxis": {
"align": false,
"alignLevel": null
}
},
{
"aliasColors": {},
"bars": false,
"dashLength": 10,
"dashes": false,
"datasource": null,
"fill": 1,
"gridPos": {
"h": 8,
"w": 8,
"x": 0,
"y": 28
},
"id": 72,
"legend": {
"avg": false,
"current": false,
"max": false,
"min": false,
"show": true,
"total": false,
"values": false
},
"lines": true,
"linewidth": 1,
"links": [],
"nullPointMode": "null",
"percentage": false,
"pointradius": 5,
"points": false,
"renderer": "flot",
"seriesOverrides": [],
"spaceLength": 10,
"stack": false,
"steppedLine": false,
"targets": [
{
"expr": "[[app]]_metanode_OpMetaCreateDentry{app=\"$app\",cluster=~\"$cluster\"}",
"format": "time_series",
"intervalFactor": 1,
"legendFormat": "OpMetaCreateDentry_{{instance}}",
"refId": "A"
},
{
"expr": "[[app]]_metanode_OpMetaDeleteDentry{app=\"$app\",cluster=~\"$cluster\"}",
"format": "time_series",
"hide": true,
"intervalFactor": 1,
"legendFormat": "OpMetaDeleteDentry_{{instance}}",
"refId": "B"
},
{
"expr": "[[app]]_metanode_OpMetaUpdateDentry{app=\"$app\",cluster=~\"$cluster\"}",
"format": "time_series",
"hide": true,
"intervalFactor": 1,
"legendFormat": "OpMetaUpdateDentry_{{instance}}",
"refId": "C"
}
],
"thresholds": [],
"timeFrom": null,
"timeShift": null,
"title": "OpMetaCreateDentry",
"tooltip": {
"shared": true,
"sort": 0,
"value_type": "individual"
},
"type": "graph",
"xaxis": {
"buckets": null,
"mode": "time",
"name": null,
"show": true,
"values": []
},
"yaxes": [
{
"format": "ns",
"label": null,
"logBase": 1,
"max": null,
"min": null,
"show": true
},
{
"format": "short",
"label": null,
"logBase": 1,
"max": null,
"min": null,
"show": true
}
],
"yaxis": {
"align": false,
"alignLevel": null
}
},
{
"aliasColors": {},
"bars": false,
"dashLength": 10,
"dashes": false,
"datasource": null,
"fill": 1,
"gridPos": {
"h": 8,
"w": 8,
"x": 8,
"y": 28
},
"id": 125,
"legend": {
"avg": false,
"current": false,
"max": false,
"min": false,
"show": true,
"total": false,
"values": false
},
"lines": true,
"linewidth": 1,
"links": [],
"nullPointMode": "null",
"percentage": false,
"pointradius": 5,
"points": false,
"renderer": "flot",
"seriesOverrides": [],
"spaceLength": 10,
"stack": false,
"steppedLine": false,
"targets": [
{
"expr": "[[app]]_metanode_OpMetaOpen{app=~\"$app\", cluster=\"$cluster\"}",
"format": "time_series",
"hide": true,
"intervalFactor": 1,
"legendFormat": "OpMetaOpen_{{instance}}",
"refId": "A"
},
{
"expr": "[[app]]_metanode_OpMetaLookup{app=~\"$app\", cluster=\"$cluster\", instance=\"$instance\"}",
"format": "time_series",
"hide": false,
"intervalFactor": 1,
"legendFormat": "OpMetaLookup_{{instance}}",
"refId": "B"
},
{
"expr": "[[app]]_metanode_OpMetaNodeHeartbeat{app=~\"$app\", cluster=\"$cluster\"}",
"format": "time_series",
"hide": true,
"intervalFactor": 1,
"legendFormat": "OpMetaNodeHeartbeat_{{instance}}",
"refId": "C"
},
{
"expr": "[[app]]_metanode_OpMetaReadDir{app=~\"$app\", cluster=\"$cluster\"}",
"format": "time_series",
"hide": true,
"intervalFactor": 1,
"legendFormat": "OpMetaReadDir_{{instance}}",
"refId": "D"
},
{
"expr": "[[app]]_metanode_OpMetaReleaseOpen{app=~\"$app\", cluster=\"$cluster\"}",
"format": "time_series",
"hide": true,
"intervalFactor": 1,
"legendFormat": "OpMetaReleaseOpen_{{instance}}",
"refId": "E"
},
{
"expr": "[[app]]_metanode_OpMetaSetattr{app=~\"$app\", cluster=\"$cluster\"}",
"format": "time_series",
"hide": true,
"intervalFactor": 1,
"legendFormat": "OpMetaSetattr_{{instance}}",
"refId": "F"
},
{
"expr": "[[app]]_metanode_OpMetaTruncate{app=~\"$app\", cluster=\"$cluster\"}",
"format": "time_series",
"hide": true,
"intervalFactor": 1,
"legendFormat": "OpMetaTruncate_{{instance}}",
"refId": "G"
}
],
"thresholds": [],
"timeFrom": null,
"timeShift": null,
"title": "OpMetaLookup",
"tooltip": {
"shared": true,
"sort": 0,
"value_type": "individual"
},
"type": "graph",
"xaxis": {
"buckets": null,
"mode": "time",
"name": null,
"show": true,
"values": []
},
"yaxes": [
{
"format": "ns",
"label": null,
"logBase": 1,
"max": null,
"min": null,
"show": true
},
{
"format": "short",
"label": null,
"logBase": 1,
"max": null,
"min": null,
"show": true
}
],
"yaxis": {
"align": false,
"alignLevel": null
}
},
{
"aliasColors": {},
"bars": false,
"dashLength": 10,
"dashes": false,
"datasource": null,
"fill": 1,
"gridPos": {
"h": 8,
"w": 8,
"x": 16,
"y": 28
},
"id": 122,
"legend": {
"avg": false,
"current": false,
"max": false,
"min": false,
"show": true,
"total": false,
"values": false
},
"lines": true,
"linewidth": 1,
"links": [],
"nullPointMode": "null",
"percentage": false,
"pointradius": 5,
"points": false,
"renderer": "flot",
"seriesOverrides": [],
"spaceLength": 10,
"stack": false,
"steppedLine": false,
"targets": [
{
"expr": "[[app]]_metanode_OpMetaCreateDentry{app=\"$app\",cluster=~\"$cluster\", instance=\"$instance\"}",
"format": "time_series",
"hide": true,
"intervalFactor": 1,
"legendFormat": "OpMetaCreateDentry_{{instance}}",
"refId": "A"
},
{
"expr": "[[app]]_metanode_OpMetaDeleteDentry{app=\"$app\",cluster=~\"$cluster\", instance=\"$instance\"}",
"format": "time_series",
"hide": false,
"intervalFactor": 1,
"legendFormat": "OpMetaDeleteDentry_{{instance}}",
"refId": "B"
},
{
"expr": "[[app]]_metanode_OpMetaUpdateDentry{app=\"$app\",cluster=~\"$cluster\", instance=\"$instance\"}",
"format": "time_series",
"hide": true,
"intervalFactor": 1,
"legendFormat": "OpMetaUpdateDentry_{{instance}}",
"refId": "C"
}
],
"thresholds": [],
"timeFrom": null,
"timeShift": null,
"title": "OpMetaDeleteDentry",
"tooltip": {
"shared": true,
"sort": 0,
"value_type": "individual"
},
"type": "graph",
"xaxis": {
"buckets": null,
"mode": "time",
"name": null,
"show": true,
"values": []
},
"yaxes": [
{
"format": "ns",
"label": null,
"logBase": 1,
"max": null,
"min": null,
"show": true
},
{
"format": "short",
"label": null,
"logBase": 1,
"max": null,
"min": null,
"show": true
}
],
"yaxis": {
"align": false,
"alignLevel": null
}
},
{
"aliasColors": {},
"bars": false,
"dashLength": 10,
"dashes": false,
"datasource": null,
"fill": 1,
"gridPos": {
"h": 8,
"w": 8,
"x": 0,
"y": 36
},
"id": 124,
"legend": {
"avg": false,
"current": false,
"max": false,
"min": false,
"show": true,
"total": false,
"values": false
},
"lines": true,
"linewidth": 1,
"links": [],
"nullPointMode": "null",
"percentage": false,
"pointradius": 5,
"points": false,
"renderer": "flot",
"seriesOverrides": [
{
"alias": "/.*_ops_.*/",
"yaxis": 2
}
],
"spaceLength": 10,
"stack": false,
"steppedLine": false,
"targets": [
{
"expr": "[[app]]_metanode_OpMetaReadDir{app=~\"$app\", cluster=\"$cluster\", instance=\"$instance\"}",
"format": "time_series",
"hide": false,
"intervalFactor": 1,
"legendFormat": "OpMetaReadDir_{{instance}}",
"refId": "D"
},
{
"expr": "rate([[app]]_metanode_OpMetaReadDir_count{app=~\"$app\", cluster=\"$cluster\", instance=\"$instance\"}[1m])",
"format": "time_series",
"hide": false,
"intervalFactor": 1,
"legendFormat": "OpMetaReadDir_ops_{{instance}}",
"refId": "A"
}
],
"thresholds": [],
"timeFrom": null,
"timeShift": null,
"title": "OpMetaReadDir",
"tooltip": {
"shared": true,
"sort": 0,
"value_type": "individual"
},
"type": "graph",
"xaxis": {
"buckets": null,
"mode": "time",
"name": null,
"show": true,
"values": []
},
"yaxes": [
{
"format": "ns",
"label": null,
"logBase": 1,
"max": null,
"min": null,
"show": true
},
{
"format": "ops",
"label": null,
"logBase": 1,
"max": null,
"min": null,
"show": true
}
],
"yaxis": {
"align": false,
"alignLevel": null
}
},
{
"aliasColors": {},
"bars": false,
"dashLength": 10,
"dashes": false,
"datasource": null,
"fill": 1,
"gridPos": {
"h": 8,
"w": 8,
"x": 8,
"y": 36
},
"id": 129,
"legend": {
"avg": false,
"current": false,
"max": false,
"min": false,
"show": true,
"total": false,
"values": false
},
"lines": true,
"linewidth": 1,
"links": [],
"nullPointMode": "null",
"percentage": false,
"pointradius": 5,
"points": false,
"renderer": "flot",
"seriesOverrides": [],
"spaceLength": 10,
"stack": false,
"steppedLine": false,
"targets": [
{
"expr": "[[app]]_metanode_OpMetaOpen{app=~\"$app\", cluster=\"$cluster\"}",
"format": "time_series",
"hide": true,
"intervalFactor": 1,
"legendFormat": "OpMetaOpen_{{instance}}",
"refId": "A"
},
{
"expr": "[[app]]_metanode_OpMetaLookup{app=~\"$app\", cluster=\"$cluster\"}",
"format": "time_series",
"hide": true,
"intervalFactor": 1,
"legendFormat": "OpMetaLookup_{{instance}}",
"refId": "B"
},
{
"expr": "[[app]]_metanode_OpMetaNodeHeartbeat{app=~\"$app\", cluster=\"$cluster\"}",
"format": "time_series",
"hide": true,
"intervalFactor": 1,
"legendFormat": "OpMetaNodeHeartbeat_{{instance}}",
"refId": "C"
},
{
"expr": "[[app]]_metanode_OpMetaReadDir{app=~\"$app\", cluster=\"$cluster\"}",
"format": "time_series",
"hide": true,
"intervalFactor": 1,
"legendFormat": "OpMetaReadDir_{{instance}}",
"refId": "D"
},
{
"expr": "[[app]]_metanode_OpMetaReleaseOpen{app=~\"$app\", cluster=\"$cluster\"}",
"format": "time_series",
"hide": false,
"intervalFactor": 1,
"legendFormat": "OpMetaReleaseOpen_{{instance}}",
"refId": "E"
},
{
"expr": "[[app]]_metanode_OpMetaSetattr{app=~\"$app\", cluster=\"$cluster\"}",
"format": "time_series",
"hide": true,
"intervalFactor": 1,
"legendFormat": "OpMetaSetattr_{{instance}}",
"refId": "F"
},
{
"expr": "[[app]]_metanode_OpMetaTruncate{app=~\"$app\", cluster=\"$cluster\"}",
"format": "time_series",
"hide": true,
"intervalFactor": 1,
"legendFormat": "OpMetaTruncate_{{instance}}",
"refId": "G"
}
],
"thresholds": [],
"timeFrom": null,
"timeShift": null,
"title": "OpMetaReleaseOpen",
"tooltip": {
"shared": true,
"sort": 0,
"value_type": "individual"
},
"type": "graph",
"xaxis": {
"buckets": null,
"mode": "time",
"name": null,
"show": true,
"values": []
},
"yaxes": [
{
"format": "ns",
"label": null,
"logBase": 1,
"max": null,
"min": null,
"show": true
},
{
"format": "short",
"label": null,
"logBase": 1,
"max": null,
"min": null,
"show": true
}
],
"yaxis": {
"align": false,
"alignLevel": null
}
},
{
"aliasColors": {},
"bars": false,
"dashLength": 10,
"dashes": false,
"datasource": null,
"fill": 1,
"gridPos": {
"h": 8,
"w": 8,
"x": 16,
"y": 36
},
"id": 116,
"legend": {
"avg": false,
"current": false,
"max": false,
"min": false,
"show": true,
"total": false,
"values": false
},
"lines": true,
"linewidth": 1,
"links": [],
"nullPointMode": "null",
"percentage": false,
"pointradius": 5,
"points": false,
"renderer": "flot",
"seriesOverrides": [],
"spaceLength": 10,
"stack": false,
"steppedLine": false,
"targets": [
{
"expr": "[[app]]_metanode_OpMetaBatchInodeGet{app=~\"$app\", cluster=\"$cluster\"}",
"format": "time_series",
"hide": true,
"intervalFactor": 1,
"legendFormat": "OpMetaBatchInodeGet_{{instance}}",
"refId": "A"
},
{
"expr": "[[app]]_metanode_OpMetaCreateInode{app=~\"$app\", cluster=\"$cluster\"}",
"format": "time_series",
"hide": true,
"intervalFactor": 1,
"legendFormat": "OpMetaCreateInode",
"refId": "B"
},
{
"expr": "[[app]]_metanode_OpMetaDeleteInode{app=~\"$app\", cluster=\"$cluster\"}",
"format": "time_series",
"hide": false,
"intervalFactor": 1,
"legendFormat": "OpMetaDeleteInode_{{instance}}",
"refId": "C"
},
{
"expr": "[[app]]_metanode_OpMetaEvictInode{app=~\"$app\", cluster=\"$cluster\"}",
"format": "time_series",
"hide": true,
"intervalFactor": 1,
"legendFormat": "OpMetaEvictInode_{{instance}}",
"refId": "D"
},
{
"expr": "[[app]]_metanode_OpMetaInodeGet{app=~\"$app\", cluster=\"$cluster\"}",
"format": "time_series",
"hide": true,
"intervalFactor": 1,
"legendFormat": "OpMetaInodeGet_{{instance}}",
"refId": "E"
},
{
"expr": "[[app]]_metanode_OpMetaLinkInode{app=~\"$app\", cluster=\"$cluster\"}",
"format": "time_series",
"hide": true,
"intervalFactor": 1,
"legendFormat": "OpMetaLinkInode_{{instance}}",
"refId": "F"
}
],
"thresholds": [],
"timeFrom": null,
"timeShift": null,
"title": "OpMetaDeleteInode",
"tooltip": {
"shared": true,
"sort": 0,
"value_type": "individual"
},
"type": "graph",
"xaxis": {
"buckets": null,
"mode": "time",
"name": null,
"show": true,
"values": []
},
"yaxes": [
{
"format": "ns",
"label": null,
"logBase": 1,
"max": null,
"min": null,
"show": true
},
{
"format": "short",
"label": null,
"logBase": 1,
"max": null,
"min": null,
"show": true
}
],
"yaxis": {
"align": false,
"alignLevel": null
}
},
{
"aliasColors": {},
"bars": false,
"dashLength": 10,
"dashes": false,
"datasource": null,
"fill": 1,
"gridPos": {
"h": 8,
"w": 8,
"x": 0,
"y": 44
},
"id": 127,
"legend": {
"avg": false,
"current": false,
"max": false,
"min": false,
"show": true,
"total": false,
"values": false
},
"lines": true,
"linewidth": 1,
"links": [],
"nullPointMode": "null",
"percentage": false,
"pointradius": 5,
"points": false,
"renderer": "flot",
"seriesOverrides": [],
"spaceLength": 10,
"stack": false,
"steppedLine": false,
"targets": [
{
"expr": "[[app]]_metanode_OpMetaOpen{app=~\"$app\", cluster=\"$cluster\"}",
"format": "time_series",
"hide": true,
"intervalFactor": 1,
"legendFormat": "OpMetaOpen_{{instance}}",
"refId": "A"
},
{
"expr": "[[app]]_metanode_OpMetaLookup{app=~\"$app\", cluster=\"$cluster\"}",
"format": "time_series",
"hide": true,
"intervalFactor": 1,
"legendFormat": "OpMetaLookup_{{instance}}",
"refId": "B"
},
{
"expr": "[[app]]_metanode_OpMetaNodeHeartbeat{app=~\"$app\", cluster=\"$cluster\"}",
"format": "time_series",
"hide": true,
"intervalFactor": 1,
"legendFormat": "OpMetaNodeHeartbeat_{{instance}}",
"refId": "C"
},
{
"expr": "[[app]]_metanode_OpMetaReadDir{app=~\"$app\", cluster=\"$cluster\"}",
"format": "time_series",
"hide": true,
"intervalFactor": 1,
"legendFormat": "OpMetaReadDir_{{instance}}",
"refId": "D"
},
{
"expr": "[[app]]_metanode_OpMetaReleaseOpen{app=~\"$app\", cluster=\"$cluster\"}",
"format": "time_series",
"hide": true,
"intervalFactor": 1,
"legendFormat": "OpMetaReleaseOpen_{{instance}}",
"refId": "E"
},
{
"expr": "[[app]]_metanode_OpMetaSetattr{app=~\"$app\", cluster=\"$cluster\"}",
"format": "time_series",
"hide": true,
"intervalFactor": 1,
"legendFormat": "OpMetaSetattr_{{instance}}",
"refId": "F"
},
{
"expr": "[[app]]_metanode_OpMetaTruncate{app=~\"$app\", cluster=\"$cluster\"}",
"format": "time_series",
"hide": false,
"intervalFactor": 1,
"legendFormat": "OpMetaTruncate_{{instance}}",
"refId": "G"
}
],
"thresholds": [],
"timeFrom": null,
"timeShift": null,
"title": "OpMetaTruncate",
"tooltip": {
"shared": true,
"sort": 0,
"value_type": "individual"
},
"type": "graph",
"xaxis": {
"buckets": null,
"mode": "time",
"name": null,
"show": true,
"values": []
},
"yaxes": [
{
"format": "ns",
"label": null,
"logBase": 1,
"max": null,
"min": null,
"show": true
},
{
"format": "short",
"label": null,
"logBase": 1,
"max": null,
"min": null,
"show": true
}
],
"yaxis": {
"align": false,
"alignLevel": null
}
},
{
"aliasColors": {},
"bars": false,
"dashLength": 10,
"dashes": false,
"datasource": null,
"fill": 1,
"gridPos": {
"h": 8,
"w": 8,
"x": 8,
"y": 44
},
"id": 128,
"legend": {
"avg": false,
"current": false,
"max": false,
"min": false,
"show": true,
"total": false,
"values": false
},
"lines": true,
"linewidth": 1,
"links": [],
"nullPointMode": "null",
"percentage": false,
"pointradius": 5,
"points": false,
"renderer": "flot",
"seriesOverrides": [],
"spaceLength": 10,
"stack": false,
"steppedLine": false,
"targets": [
{
"expr": "[[app]]_metanode_OpMetaOpen{app=~\"$app\", cluster=\"$cluster\"}",
"format": "time_series",
"hide": true,
"intervalFactor": 1,
"legendFormat": "OpMetaOpen_{{instance}}",
"refId": "A"
},
{
"expr": "[[app]]_metanode_OpMetaLookup{app=~\"$app\", cluster=\"$cluster\"}",
"format": "time_series",
"hide": true,
"intervalFactor": 1,
"legendFormat": "OpMetaLookup_{{instance}}",
"refId": "B"
},
{
"expr": "[[app]]_metanode_OpMetaNodeHeartbeat{app=~\"$app\", cluster=\"$cluster\"}",
"format": "time_series",
"hide": true,
"intervalFactor": 1,
"legendFormat": "OpMetaNodeHeartbeat_{{instance}}",
"refId": "C"
},
{
"expr": "[[app]]_metanode_OpMetaReadDir{app=~\"$app\", cluster=\"$cluster\"}",
"format": "time_series",
"hide": true,
"intervalFactor": 1,
"legendFormat": "OpMetaReadDir_{{instance}}",
"refId": "D"
},
{
"expr": "[[app]]_metanode_OpMetaReleaseOpen{app=~\"$app\", cluster=\"$cluster\"}",
"format": "time_series",
"hide": true,
"intervalFactor": 1,
"legendFormat": "OpMetaReleaseOpen_{{instance}}",
"refId": "E"
},
{
"expr": "[[app]]_metanode_OpMetaSetattr{app=~\"$app\", cluster=\"$cluster\"}",
"format": "time_series",
"hide": false,
"intervalFactor": 1,
"legendFormat": "OpMetaSetattr_{{instance}}",
"refId": "F"
},
{
"expr": "[[app]]_metanode_OpMetaTruncate{app=~\"$app\", cluster=\"$cluster\"}",
"format": "time_series",
"hide": true,
"intervalFactor": 1,
"legendFormat": "OpMetaTruncate_{{instance}}",
"refId": "G"
}
],
"thresholds": [],
"timeFrom": null,
"timeShift": null,
"title": "OpMetaSetattr",
"tooltip": {
"shared": true,
"sort": 0,
"value_type": "individual"
},
"type": "graph",
"xaxis": {
"buckets": null,
"mode": "time",
"name": null,
"show": true,
"values": []
},
"yaxes": [
{
"format": "ns",
"label": null,
"logBase": 1,
"max": null,
"min": null,
"show": true
},
{
"format": "short",
"label": null,
"logBase": 1,
"max": null,
"min": null,
"show": true
}
],
"yaxis": {
"align": false,
"alignLevel": null
}
},
{
"aliasColors": {},
"bars": false,
"dashLength": 10,
"dashes": false,
"datasource": null,
"fill": 1,
"gridPos": {
"h": 8,
"w": 8,
"x": 16,
"y": 44
},
"id": 123,
"legend": {
"avg": false,
"current": false,
"max": false,
"min": false,
"show": true,
"total": false,
"values": false
},
"lines": true,
"linewidth": 1,
"links": [],
"nullPointMode": "null",
"percentage": false,
"pointradius": 5,
"points": false,
"renderer": "flot",
"seriesOverrides": [],
"spaceLength": 10,
"stack": false,
"steppedLine": false,
"targets": [
{
"expr": "[[app]]_metanode_OpMetaCreateDentry{app=\"$app\",cluster=~\"$cluster\"}",
"format": "time_series",
"hide": true,
"intervalFactor": 1,
"legendFormat": "OpMetaCreateDentry_{{instance}}",
"refId": "A"
},
{
"expr": "[[app]]_metanode_OpMetaDeleteDentry{app=\"$app\",cluster=~\"$cluster\"}",
"format": "time_series",
"hide": true,
"intervalFactor": 1,
"legendFormat": "OpMetaDeleteDentry_{{instance}}",
"refId": "B"
},
{
"expr": "[[app]]_metanode_OpMetaUpdateDentry{app=\"$app\",cluster=~\"$cluster\"}",
"format": "time_series",
"hide": false,
"intervalFactor": 1,
"legendFormat": "OpMetaUpdateDentry_{{instance}}",
"refId": "C"
}
],
"thresholds": [],
"timeFrom": null,
"timeShift": null,
"title": "OpMetaUpdateDentry",
"tooltip": {
"shared": true,
"sort": 0,
"value_type": "individual"
},
"type": "graph",
"xaxis": {
"buckets": null,
"mode": "time",
"name": null,
"show": true,
"values": []
},
"yaxes": [
{
"format": "ns",
"label": null,
"logBase": 1,
"max": null,
"min": null,
"show": true
},
{
"format": "short",
"label": null,
"logBase": 1,
"max": null,
"min": null,
"show": true
}
],
"yaxis": {
"align": false,
"alignLevel": null
}
},
{
"aliasColors": {},
"bars": false,
"dashLength": 10,
"dashes": false,
"datasource": null,
"fill": 1,
"gridPos": {
"h": 8,
"w": 8,
"x": 0,
"y": 52
},
"id": 46,
"legend": {
"avg": false,
"current": false,
"max": false,
"min": false,
"show": true,
"total": false,
"values": false
},
"lines": true,
"linewidth": 1,
"links": [],
"nullPointMode": "null",
"percentage": false,
"pointradius": 5,
"points": false,
"renderer": "flot",
"seriesOverrides": [],
"spaceLength": 10,
"stack": false,
"steppedLine": false,
"targets": [
{
"expr": "[[app]]_metanode_OpMetaOpen{app=~\"$app\", cluster=\"$cluster\"}",
"format": "time_series",
"intervalFactor": 1,
"legendFormat": "OpMetaOpen_{{instance}}",
"refId": "A"
},
{
"expr": "[[app]]_metanode_OpMetaLookup{app=~\"$app\", cluster=\"$cluster\"}",
"format": "time_series",
"hide": true,
"intervalFactor": 1,
"legendFormat": "OpMetaLookup_{{instance}}",
"refId": "B"
},
{
"expr": "[[app]]_metanode_OpMetaNodeHeartbeat{app=~\"$app\", cluster=\"$cluster\"}",
"format": "time_series",
"hide": true,
"intervalFactor": 1,
"legendFormat": "OpMetaNodeHeartbeat_{{instance}}",
"refId": "C"
},
{
"expr": "[[app]]_metanode_OpMetaReadDir{app=~\"$app\", cluster=\"$cluster\"}",
"format": "time_series",
"hide": true,
"intervalFactor": 1,
"legendFormat": "OpMetaReadDir_{{instance}}",
"refId": "D"
},
{
"expr": "[[app]]_metanode_OpMetaReleaseOpen{app=~\"$app\", cluster=\"$cluster\"}",
"format": "time_series",
"hide": true,
"intervalFactor": 1,
"legendFormat": "OpMetaReleaseOpen_{{instance}}",
"refId": "E"
},
{
"expr": "[[app]]_metanode_OpMetaSetattr{app=~\"$app\", cluster=\"$cluster\"}",
"format": "time_series",
"hide": true,
"intervalFactor": 1,
"legendFormat": "OpMetaSetattr_{{instance}}",
"refId": "F"
},
{
"expr": "[[app]]_metanode_OpMetaTruncate{app=~\"$app\", cluster=\"$cluster\"}",
"format": "time_series",
"hide": true,
"intervalFactor": 1,
"legendFormat": "OpMetaTruncate_{{instance}}",
"refId": "G"
}
],
"thresholds": [],
"timeFrom": null,
"timeShift": null,
"title": "OpMetaOpen",
"tooltip": {
"shared": true,
"sort": 0,
"value_type": "individual"
},
"type": "graph",
"xaxis": {
"buckets": null,
"mode": "time",
"name": null,
"show": true,
"values": []
},
"yaxes": [
{
"format": "ns",
"label": null,
"logBase": 1,
"max": null,
"min": null,
"show": true
},
{
"format": "short",
"label": null,
"logBase": 1,
"max": null,
"min": null,
"show": true
}
],
"yaxis": {
"align": false,
"alignLevel": null
}
}
],
"title": "Metanode",
"type": "row"
},
{
"collapsed": true,
"gridPos": {
"h": 1,
"w": 24,
"x": 0,
"y": 37
},
"id": 27,
"panels": [
{
"aliasColors": {},
"bars": false,
"dashLength": 10,
"dashes": false,
"datasource": null,
"fill": 1,
"gridPos": {
"h": 8,
"w": 8,
"x": 0,
"y": 5
},
"id": 132,
"legend": {
"avg": false,
"current": false,
"max": false,
"min": false,
"show": true,
"total": false,
"values": false
},
"lines": true,
"linewidth": 1,
"links": [],
"nullPointMode": "null",
"percentage": false,
"pointradius": 5,
"points": false,
"renderer": "flot",
"seriesOverrides": [
{
"alias": "/.*_ops_.*/",
"yaxis": 2
}
],
"spaceLength": 10,
"stack": false,
"steppedLine": false,
"targets": [
{
"expr": "[[app]]_dataNode_OpDataNodeHeartbeat{app=~\"$app\", cluster=\"$cluster\"}",
"format": "time_series",
"hide": false,
"intervalFactor": 1,
"legendFormat": "DataNodeHeartbeat_{{instance}}",
"refId": "D"
},
{
"expr": "rate([[app]]_dataNode_OpDataNodeHeartbeat{app=~\"$app\", cluster=\"$cluster\"}[1m])",
"format": "time_series",
"hide": false,
"intervalFactor": 1,
"legendFormat": "DataNodeHeartbeat_ops_{{instance}}",
"refId": "Q"
}
],
"thresholds": [],
"timeFrom": null,
"timeShift": null,
"title": "DataNodeHeartbeat",
"tooltip": {
"shared": true,
"sort": 0,
"value_type": "individual"
},
"type": "graph",
"xaxis": {
"buckets": null,
"mode": "time",
"name": null,
"show": true,
"values": []
},
"yaxes": [
{
"format": "ns",
"label": null,
"logBase": 1,
"max": null,
"min": null,
"show": true
},
{
"format": "ops",
"label": null,
"logBase": 1,
"max": null,
"min": null,
"show": true
}
],
"yaxis": {
"align": false,
"alignLevel": null
}
},
{
"aliasColors": {},
"bars": false,
"dashLength": 10,
"dashes": false,
"datasource": null,
"fill": 1,
"gridPos": {
"h": 8,
"w": 8,
"x": 8,
"y": 5
},
"id": 133,
"legend": {
"avg": false,
"current": false,
"max": false,
"min": false,
"show": true,
"total": false,
"values": false
},
"lines": true,
"linewidth": 1,
"links": [],
"nullPointMode": "null",
"percentage": false,
"pointradius": 5,
"points": false,
"renderer": "flot",
"seriesOverrides": [],
"spaceLength": 10,
"stack": false,
"steppedLine": false,
"targets": [
{
"expr": "[[app]]_dataNode_OpBroadcastMinAppliedID{app=~\"$app\", cluster=\"$cluster\"}",
"format": "time_series",
"hide": true,
"intervalFactor": 1,
"legendFormat": "OpBroadcastMinAppliedID_{{instance}}",
"refId": "B"
},
{
"expr": "[[app]]_dataNode_OpCreateDataPartition{app=~\"$app\", cluster=\"$cluster\"}",
"format": "time_series",
"hide": false,
"intervalFactor": 1,
"legendFormat": "OpCreateDataPartition_{{instance}}",
"refId": "C"
},
{
"expr": "[[app]]_dataNode_OpDataNodeHeartbeat{app=~\"$app\", cluster=\"$cluster\"}",
"format": "time_series",
"hide": true,
"intervalFactor": 1,
"legendFormat": "DataNodeHeartbeat_{{instance}}",
"refId": "D"
},
{
"expr": "[[app]]_dataNode_OpDecommissionDataPartition{app=~\"$app\", cluster=\"$cluster\"}",
"format": "time_series",
"hide": true,
"intervalFactor": 1,
"legendFormat": "DecommissionDataPartition_{{instance}}",
"refId": "A"
},
{
"expr": "[[app]]_dataNode_OpGetAllWatermarks{app=~\"$app\", cluster=\"$cluster\"}",
"format": "time_series",
"hide": true,
"intervalFactor": 1,
"legendFormat": "GetAllWatermarks_{{instance}}",
"refId": "E"
},
{
"expr": "[[app]]_dataNode_OpGetAppliedId{app=~\"$app\", cluster=\"$cluster\"}",
"format": "time_series",
"hide": true,
"intervalFactor": 1,
"legendFormat": "GetAppliedId_{{instance}}",
"refId": "F"
},
{
"expr": "[[app]]_dataNode_OpGetPartitionSize{app=~\"$app\", cluster=\"$cluster\"}",
"format": "time_series",
"hide": true,
"intervalFactor": 1,
"legendFormat": "GetPartitionSize_{{instance}}",
"refId": "G"
},
{
"expr": "[[app]]_dataNode_OpLoadDataPartition{app=~\"$app\", cluster=\"$cluster\"}",
"format": "time_series",
"hide": true,
"intervalFactor": 1,
"legendFormat": "LoadDataPartition_{{instance}}",
"refId": "H"
},
{
"expr": "[[app]]_dataNode_OpMarkDelete{app=~\"$app\", cluster=\"$cluster\"}",
"format": "time_series",
"hide": true,
"intervalFactor": 1,
"legendFormat": "MarkDelete_{{instance}}",
"refId": "I"
},
{
"expr": "[[app]]_dataNode_OpNotifyReplicasToRepair{app=~\"$app\", cluster=\"$cluster\"}",
"format": "time_series",
"hide": true,
"intervalFactor": 1,
"legendFormat": "NotifyReplicasToRepair_{{instance}}",
"refId": "J"
},
{
"expr": "[[app]]_dataNode_OpRandomWrite{app=~\"$app\", cluster=\"$cluster\"}",
"format": "time_series",
"hide": true,
"intervalFactor": 1,
"legendFormat": "OpRandomWrite_{{instance}}",
"refId": "K"
},
{
"expr": "[[app]]_dataNode_OpReadTinyDelete{app=~\"$app\", cluster=\"$cluster\"}",
"format": "time_series",
"hide": true,
"intervalFactor": 1,
"legendFormat": "ReadTinyDelete_{{instance}}",
"refId": "L"
},
{
"expr": "[[app]]_dataNode_OpStreamRead{app=~\"$app\", cluster=\"$cluster\"}",
"format": "time_series",
"hide": true,
"intervalFactor": 1,
"legendFormat": "StreamRead_{{instance}}",
"refId": "M"
},
{
"expr": "[[app]]_dataNode_OpWrite{app=~\"$app\", cluster=\"$cluster\"}",
"format": "time_series",
"hide": true,
"intervalFactor": 1,
"legendFormat": "Write_{{instance}}",
"refId": "N"
},
{
"expr": "[[app]]_dataNode_OpCreateExtent{app=~\"$app\", cluster=\"$cluster\"}",
"format": "time_series",
"hide": true,
"intervalFactor": 1,
"legendFormat": "CreateExtent_{{instance}}",
"refId": "O"
},
{
"expr": "[[app]]_dataNode_OpExtentRepairRead{app=~\"$app\", cluster=\"$cluster\"}",
"format": "time_series",
"hide": true,
"intervalFactor": 1,
"legendFormat": "ExtentRepairdRead_{{instance}}",
"refId": "P"
}
],
"thresholds": [],
"timeFrom": null,
"timeShift": null,
"title": "OpCreateDataPartition",
"tooltip": {
"shared": true,
"sort": 0,
"value_type": "individual"
},
"type": "graph",
"xaxis": {
"buckets": null,
"mode": "time",
"name": null,
"show": true,
"values": []
},
"yaxes": [
{
"format": "ns",
"label": null,
"logBase": 1,
"max": null,
"min": null,
"show": true
},
{
"format": "short",
"label": null,
"logBase": 1,
"max": null,
"min": null,
"show": true
}
],
"yaxis": {
"align": false,
"alignLevel": null
}
},
{
"aliasColors": {},
"bars": false,
"dashLength": 10,
"dashes": false,
"datasource": null,
"fill": 1,
"gridPos": {
"h": 8,
"w": 8,
"x": 16,
"y": 5
},
"id": 137,
"legend": {
"avg": false,
"current": false,
"max": false,
"min": false,
"show": true,
"total": false,
"values": false
},
"lines": true,
"linewidth": 1,
"links": [],
"nullPointMode": "null",
"percentage": false,
"pointradius": 5,
"points": false,
"renderer": "flot",
"seriesOverrides": [
{
"alias": "/.*_ops_.*/",
"yaxis": 2
}
],
"spaceLength": 10,
"stack": false,
"steppedLine": false,
"targets": [
{
"expr": "[[app]]_dataNode_OpLoadDataPartition{app=~\"$app\", cluster=\"$cluster\"}",
"format": "time_series",
"hide": false,
"intervalFactor": 1,
"legendFormat": "LoadDataPartition_{{instance}}",
"refId": "H"
},
{
"expr": "rate([[app]]_dataNode_OpLoadDataPartition_count{app=~\"$app\", cluster=\"$cluster\"}[1m])",
"format": "time_series",
"hide": false,
"intervalFactor": 1,
"legendFormat": "LoadDataPartition_ops_{{instance}}",
"refId": "A"
}
],
"thresholds": [],
"timeFrom": null,
"timeShift": null,
"title": "LoadDataPartition",
"tooltip": {
"shared": true,
"sort": 0,
"value_type": "individual"
},
"type": "graph",
"xaxis": {
"buckets": null,
"mode": "time",
"name": null,
"show": true,
"values": []
},
"yaxes": [
{
"format": "ns",
"label": null,
"logBase": 1,
"max": null,
"min": null,
"show": true
},
{
"format": "ops",
"label": null,
"logBase": 1,
"max": null,
"min": null,
"show": true
}
],
"yaxis": {
"align": false,
"alignLevel": null
}
},
{
"aliasColors": {},
"bars": false,
"dashLength": 10,
"dashes": false,
"datasource": null,
"fill": 1,
"gridPos": {
"h": 8,
"w": 8,
"x": 0,
"y": 13
},
"id": 138,
"legend": {
"avg": false,
"current": false,
"max": false,
"min": false,
"show": true,
"total": false,
"values": false
},
"lines": true,
"linewidth": 1,
"links": [],
"nullPointMode": "null",
"percentage": false,
"pointradius": 5,
"points": false,
"renderer": "flot",
"seriesOverrides": [],
"spaceLength": 10,
"stack": false,
"steppedLine": false,
"targets": [
{
"expr": "[[app]]_dataNode_OpBroadcastMinAppliedID{app=~\"$app\", cluster=\"$cluster\"}",
"format": "time_series",
"hide": true,
"intervalFactor": 1,
"legendFormat": "OpBroadcastMinAppliedID_{{instance}}",
"refId": "B"
},
{
"expr": "[[app]]_dataNode_OpCreateDataPartition{app=~\"$app\", cluster=\"$cluster\"}",
"format": "time_series",
"hide": true,
"intervalFactor": 1,
"legendFormat": "CreateDataPartition_{{instance}}",
"refId": "C"
},
{
"expr": "[[app]]_dataNode_OpDataNodeHeartbeat{app=~\"$app\", cluster=\"$cluster\"}",
"format": "time_series",
"hide": true,
"intervalFactor": 1,
"legendFormat": "DataNodeHeartbeat_{{instance}}",
"refId": "D"
},
{
"expr": "[[app]]_dataNode_OpDecommissionDataPartition{app=~\"$app\", cluster=\"$cluster\"}",
"format": "time_series",
"hide": true,
"intervalFactor": 1,
"legendFormat": "DecommissionDataPartition_{{instance}}",
"refId": "A"
},
{
"expr": "[[app]]_dataNode_OpGetAllWatermarks{app=~\"$app\", cluster=\"$cluster\"}",
"format": "time_series",
"hide": true,
"intervalFactor": 1,
"legendFormat": "GetAllWatermarks_{{instance}}",
"refId": "E"
},
{
"expr": "[[app]]_dataNode_OpGetAppliedId{app=~\"$app\", cluster=\"$cluster\"}",
"format": "time_series",
"hide": true,
"intervalFactor": 1,
"legendFormat": "GetAppliedId_{{instance}}",
"refId": "F"
},
{
"expr": "[[app]]_dataNode_OpGetPartitionSize{app=~\"$app\", cluster=\"$cluster\"}",
"format": "time_series",
"hide": true,
"intervalFactor": 1,
"legendFormat": "GetPartitionSize_{{instance}}",
"refId": "G"
},
{
"expr": "[[app]]_dataNode_OpLoadDataPartition{app=~\"$app\", cluster=\"$cluster\"}",
"format": "time_series",
"hide": true,
"intervalFactor": 1,
"legendFormat": "LoadDataPartition_{{instance}}",
"refId": "H"
},
{
"expr": "[[app]]_dataNode_OpMarkDelete{app=~\"$app\", cluster=\"$cluster\"}",
"format": "time_series",
"hide": true,
"intervalFactor": 1,
"legendFormat": "MarkDelete_{{instance}}",
"refId": "I"
},
{
"expr": "[[app]]_dataNode_OpNotifyReplicasToRepair{app=~\"$app\", cluster=\"$cluster\"}",
"format": "time_series",
"hide": false,
"intervalFactor": 1,
"legendFormat": "NotifyReplicasToRepair_{{instance}}",
"refId": "J"
},
{
"expr": "[[app]]_dataNode_OpRandomWrite{app=~\"$app\", cluster=\"$cluster\"}",
"format": "time_series",
"hide": true,
"intervalFactor": 1,
"legendFormat": "OpRandomWrite_{{instance}}",
"refId": "K"
},
{
"expr": "[[app]]_dataNode_OpReadTinyDelete{app=~\"$app\", cluster=\"$cluster\"}",
"format": "time_series",
"hide": true,
"intervalFactor": 1,
"legendFormat": "ReadTinyDelete_{{instance}}",
"refId": "L"
},
{
"expr": "[[app]]_dataNode_OpStreamRead{app=~\"$app\", cluster=\"$cluster\"}",
"format": "time_series",
"hide": true,
"intervalFactor": 1,
"legendFormat": "StreamRead_{{instance}}",
"refId": "M"
},
{
"expr": "[[app]]_dataNode_OpWrite{app=~\"$app\", cluster=\"$cluster\"}",
"format": "time_series",
"hide": true,
"intervalFactor": 1,
"legendFormat": "Write_{{instance}}",
"refId": "N"
},
{
"expr": "[[app]]_dataNode_OpCreateExtent{app=~\"$app\", cluster=\"$cluster\"}",
"format": "time_series",
"hide": true,
"intervalFactor": 1,
"legendFormat": "CreateExtent_{{instance}}",
"refId": "O"
},
{
"expr": "[[app]]_dataNode_OpExtentRepairRead{app=~\"$app\", cluster=\"$cluster\"}",
"format": "time_series",
"hide": true,
"intervalFactor": 1,
"legendFormat": "ExtentRepairdRead_{{instance}}",
"refId": "P"
}
],
"thresholds": [],
"timeFrom": null,
"timeShift": null,
"title": "NotifyReplicasToRepair",
"tooltip": {
"shared": true,
"sort": 0,
"value_type": "individual"
},
"type": "graph",
"xaxis": {
"buckets": null,
"mode": "time",
"name": null,
"show": true,
"values": []
},
"yaxes": [
{
"format": "ns",
"label": null,
"logBase": 1,
"max": null,
"min": null,
"show": true
},
{
"format": "short",
"label": null,
"logBase": 1,
"max": null,
"min": null,
"show": true
}
],
"yaxis": {
"align": false,
"alignLevel": null
}
},
{
"aliasColors": {},
"bars": false,
"dashLength": 10,
"dashes": false,
"datasource": null,
"fill": 1,
"gridPos": {
"h": 8,
"w": 8,
"x": 8,
"y": 13
},
"id": 134,
"legend": {
"avg": false,
"current": false,
"max": false,
"min": false,
"show": true,
"total": false,
"values": false
},
"lines": true,
"linewidth": 1,
"links": [],
"nullPointMode": "null",
"percentage": false,
"pointradius": 5,
"points": false,
"renderer": "flot",
"seriesOverrides": [
{
"alias": "/.*_ops_.*/",
"yaxis": 2
}
],
"spaceLength": 10,
"stack": false,
"steppedLine": false,
"targets": [
{
"expr": "[[app]]_dataNode_OpGetAllWatermarks{app=~\"$app\", cluster=\"$cluster\"}",
"format": "time_series",
"hide": false,
"intervalFactor": 1,
"legendFormat": "GetAllWatermarks_{{instance}}",
"refId": "E"
},
{
"expr": "rate([[app]]_dataNode_OpGetAllWatermarks_count{app=~\"$app\", cluster=\"$cluster\"}[1m])",
"format": "time_series",
"hide": false,
"intervalFactor": 1,
"legendFormat": "GetAllWatermarks_ops_{{instance}}",
"refId": "A"
}
],
"thresholds": [],
"timeFrom": null,
"timeShift": null,
"title": "GetAllWatermarks",
"tooltip": {
"shared": true,
"sort": 0,
"value_type": "individual"
},
"type": "graph",
"xaxis": {
"buckets": null,
"mode": "time",
"name": null,
"show": true,
"values": []
},
"yaxes": [
{
"format": "ns",
"label": null,
"logBase": 1,
"max": null,
"min": null,
"show": true
},
{
"format": "ops",
"label": null,
"logBase": 1,
"max": null,
"min": null,
"show": true
}
],
"yaxis": {
"align": false,
"alignLevel": null
}
},
{
"aliasColors": {},
"bars": false,
"dashLength": 10,
"dashes": false,
"datasource": null,
"fill": 1,
"gridPos": {
"h": 8,
"w": 8,
"x": 16,
"y": 13
},
"id": 135,
"legend": {
"avg": false,
"current": false,
"max": false,
"min": false,
"show": true,
"total": false,
"values": false
},
"lines": true,
"linewidth": 1,
"links": [],
"nullPointMode": "null",
"percentage": false,
"pointradius": 5,
"points": false,
"renderer": "flot",
"seriesOverrides": [
{
"alias": "/.*_ops_.*/",
"yaxis": 2
}
],
"spaceLength": 10,
"stack": false,
"steppedLine": false,
"targets": [
{
"expr": "[[app]]_dataNode_OpGetAppliedId{app=~\"$app\", cluster=\"$cluster\"}",
"format": "time_series",
"hide": false,
"intervalFactor": 1,
"legendFormat": "GetAppliedId_{{instance}}",
"refId": "F"
},
{
"expr": "rate([[app]]_dataNode_OpGetAppliedId_count{app=~\"$app\", cluster=\"$cluster\"}[1m])",
"format": "time_series",
"hide": false,
"intervalFactor": 1,
"legendFormat": "GetAppliedId_ops_{{instance}}",
"refId": "Q"
}
],
"thresholds": [],
"timeFrom": null,
"timeShift": null,
"title": "GetAppliedId",
"tooltip": {
"shared": true,
"sort": 0,
"value_type": "individual"
},
"type": "graph",
"xaxis": {
"buckets": null,
"mode": "time",
"name": null,
"show": true,
"values": []
},
"yaxes": [
{
"format": "ns",
"label": null,
"logBase": 1,
"max": null,
"min": null,
"show": true
},
{
"format": "ops",
"label": null,
"logBase": 1,
"max": null,
"min": null,
"show": true
}
],
"yaxis": {
"align": false,
"alignLevel": null
}
},
{
"aliasColors": {},
"bars": false,
"dashLength": 10,
"dashes": false,
"datasource": null,
"fill": 1,
"gridPos": {
"h": 8,
"w": 8,
"x": 0,
"y": 21
},
"id": 142,
"legend": {
"avg": false,
"current": false,
"max": false,
"min": false,
"show": true,
"total": false,
"values": false
},
"lines": true,
"linewidth": 1,
"links": [],
"nullPointMode": "null",
"percentage": false,
"pointradius": 5,
"points": false,
"renderer": "flot",
"seriesOverrides": [
{
"alias": "/.*_ops_.*/",
"yaxis": 2
}
],
"spaceLength": 10,
"stack": false,
"steppedLine": false,
"targets": [
{
"expr": "[[app]]_dataNode_OpStreamRead{app=~\"$app\", cluster=\"$cluster\"}",
"format": "time_series",
"hide": false,
"intervalFactor": 1,
"legendFormat": "StreamRead_{{instance}}",
"refId": "M"
},
{
"expr": "rate([[app]]dataNode_OpStreamRead_count{app=~\"$app\", cluster=\"$cluster\"}[1m])",
"format": "time_series",
"hide": false,
"intervalFactor": 1,
"legendFormat": "StreamRead_ops_{{instance}}",
"refId": "A"
}
],
"thresholds": [],
"timeFrom": null,
"timeShift": null,
"title": "StreamRead",
"tooltip": {
"shared": true,
"sort": 0,
"value_type": "individual"
},
"type": "graph",
"xaxis": {
"buckets": null,
"mode": "time",
"name": null,
"show": true,
"values": []
},
"yaxes": [
{
"format": "ns",
"label": null,
"logBase": 1,
"max": null,
"min": null,
"show": true
},
{
"format": "ops",
"label": null,
"logBase": 1,
"max": null,
"min": null,
"show": true
}
],
"yaxis": {
"align": false,
"alignLevel": null
}
},
{
"aliasColors": {},
"bars": false,
"dashLength": 10,
"dashes": false,
"datasource": null,
"fill": 1,
"gridPos": {
"h": 8,
"w": 8,
"x": 8,
"y": 21
},
"id": 143,
"legend": {
"avg": false,
"current": false,
"max": false,
"min": false,
"show": true,
"total": false,
"values": false
},
"lines": true,
"linewidth": 1,
"links": [],
"nullPointMode": "null",
"percentage": false,
"pointradius": 5,
"points": false,
"renderer": "flot",
"seriesOverrides": [],
"spaceLength": 10,
"stack": false,
"steppedLine": false,
"targets": [
{
"expr": "[[app]]_dataNode_OpWrite{app=~\"$app\", cluster=\"$cluster\"}",
"format": "time_series",
"hide": false,
"intervalFactor": 1,
"legendFormat": "Write_{{instance}}",
"refId": "N"
},
{
"expr": "[[app]]_dataNode_OpCreateExtent{app=~\"$app\", cluster=\"$cluster\"}",
"format": "time_series",
"hide": true,
"intervalFactor": 1,
"legendFormat": "CreateExtent_{{instance}}",
"refId": "O"
},
{
"expr": "[[app]]_dataNode_OpExtentRepairRead{app=~\"$app\", cluster=\"$cluster\"}",
"format": "time_series",
"hide": true,
"intervalFactor": 1,
"legendFormat": "ExtentRepairdRead_{{instance}}",
"refId": "P"
}
],
"thresholds": [],
"timeFrom": null,
"timeShift": null,
"title": "Write",
"tooltip": {
"shared": true,
"sort": 0,
"value_type": "individual"
},
"type": "graph",
"xaxis": {
"buckets": null,
"mode": "time",
"name": null,
"show": true,
"values": []
},
"yaxes": [
{
"format": "ns",
"label": null,
"logBase": 1,
"max": null,
"min": null,
"show": true
},
{
"format": "short",
"label": null,
"logBase": 1,
"max": null,
"min": null,
"show": true
}
],
"yaxis": {
"align": false,
"alignLevel": null
}
},
{
"aliasColors": {},
"bars": false,
"dashLength": 10,
"dashes": false,
"datasource": null,
"fill": 1,
"gridPos": {
"h": 8,
"w": 8,
"x": 16,
"y": 21
},
"id": 139,
"legend": {
"avg": false,
"current": false,
"max": false,
"min": false,
"show": true,
"total": false,
"values": false
},
"lines": true,
"linewidth": 1,
"links": [],
"nullPointMode": "null",
"percentage": false,
"pointradius": 5,
"points": false,
"renderer": "flot",
"seriesOverrides": [],
"spaceLength": 10,
"stack": false,
"steppedLine": false,
"targets": [
{
"expr": "[[app]]_dataNode_OpBroadcastMinAppliedID{app=~\"$app\", cluster=\"$cluster\"}",
"format": "time_series",
"hide": true,
"intervalFactor": 1,
"legendFormat": "OpBroadcastMinAppliedID_{{instance}}",
"refId": "B"
},
{
"expr": "[[app]]_dataNode_OpCreateDataPartition{app=~\"$app\", cluster=\"$cluster\"}",
"format": "time_series",
"hide": true,
"intervalFactor": 1,
"legendFormat": "CreateDataPartition_{{instance}}",
"refId": "C"
},
{
"expr": "[[app]]_dataNode_OpDataNodeHeartbeat{app=~\"$app\", cluster=\"$cluster\"}",
"format": "time_series",
"hide": true,
"intervalFactor": 1,
"legendFormat": "DataNodeHeartbeat_{{instance}}",
"refId": "D"
},
{
"expr": "[[app]]_dataNode_OpDecommissionDataPartition{app=~\"$app\", cluster=\"$cluster\"}",
"format": "time_series",
"hide": true,
"intervalFactor": 1,
"legendFormat": "DecommissionDataPartition_{{instance}}",
"refId": "A"
},
{
"expr": "[[app]]_dataNode_OpGetAllWatermarks{app=~\"$app\", cluster=\"$cluster\"}",
"format": "time_series",
"hide": true,
"intervalFactor": 1,
"legendFormat": "GetAllWatermarks_{{instance}}",
"refId": "E"
},
{
"expr": "[[app]]_dataNode_OpGetAppliedId{app=~\"$app\", cluster=\"$cluster\"}",
"format": "time_series",
"hide": true,
"intervalFactor": 1,
"legendFormat": "GetAppliedId_{{instance}}",
"refId": "F"
},
{
"expr": "[[app]]_dataNode_OpGetPartitionSize{app=~\"$app\", cluster=\"$cluster\"}",
"format": "time_series",
"hide": true,
"intervalFactor": 1,
"legendFormat": "GetPartitionSize_{{instance}}",
"refId": "G"
},
{
"expr": "[[app]]_dataNode_OpLoadDataPartition{app=~\"$app\", cluster=\"$cluster\"}",
"format": "time_series",
"hide": true,
"intervalFactor": 1,
"legendFormat": "LoadDataPartition_{{instance}}",
"refId": "H"
},
{
"expr": "[[app]]_dataNode_OpMarkDelete{app=~\"$app\", cluster=\"$cluster\"}",
"format": "time_series",
"hide": false,
"intervalFactor": 1,
"legendFormat": "MarkDelete_{{instance}}",
"refId": "I"
},
{
"expr": "[[app]]_dataNode_OpNotifyReplicasToRepair{app=~\"$app\", cluster=\"$cluster\"}",
"format": "time_series",
"hide": true,
"intervalFactor": 1,
"legendFormat": "NotifyReplicasToRepair_{{instance}}",
"refId": "J"
},
{
"expr": "[[app]]_dataNode_OpRandomWrite{app=~\"$app\", cluster=\"$cluster\"}",
"format": "time_series",
"hide": true,
"intervalFactor": 1,
"legendFormat": "OpRandomWrite_{{instance}}",
"refId": "K"
},
{
"expr": "[[app]]_dataNode_OpReadTinyDelete{app=~\"$app\", cluster=\"$cluster\"}",
"format": "time_series",
"hide": true,
"intervalFactor": 1,
"legendFormat": "ReadTinyDelete_{{instance}}",
"refId": "L"
},
{
"expr": "[[app]]_dataNode_OpStreamRead{app=~\"$app\", cluster=\"$cluster\"}",
"format": "time_series",
"hide": true,
"intervalFactor": 1,
"legendFormat": "StreamRead_{{instance}}",
"refId": "M"
},
{
"expr": "[[app]]_dataNode_OpWrite{app=~\"$app\", cluster=\"$cluster\"}",
"format": "time_series",
"hide": true,
"intervalFactor": 1,
"legendFormat": "Write_{{instance}}",
"refId": "N"
},
{
"expr": "[[app]]_dataNode_OpCreateExtent{app=~\"$app\", cluster=\"$cluster\"}",
"format": "time_series",
"hide": true,
"intervalFactor": 1,
"legendFormat": "CreateExtent_{{instance}}",
"refId": "O"
},
{
"expr": "[[app]]_dataNode_OpExtentRepairRead{app=~\"$app\", cluster=\"$cluster\"}",
"format": "time_series",
"hide": true,
"intervalFactor": 1,
"legendFormat": "ExtentRepairdRead_{{instance}}",
"refId": "P"
}
],
"thresholds": [],
"timeFrom": null,
"timeShift": null,
"title": "MarkDelete",
"tooltip": {
"shared": true,
"sort": 0,
"value_type": "individual"
},
"type": "graph",
"xaxis": {
"buckets": null,
"mode": "time",
"name": null,
"show": true,
"values": []
},
"yaxes": [
{
"format": "ns",
"label": null,
"logBase": 1,
"max": null,
"min": null,
"show": true
},
{
"format": "short",
"label": null,
"logBase": 1,
"max": null,
"min": null,
"show": true
}
],
"yaxis": {
"align": false,
"alignLevel": null
}
},
{
"aliasColors": {},
"bars": false,
"dashLength": 10,
"dashes": false,
"datasource": null,
"fill": 1,
"gridPos": {
"h": 8,
"w": 8,
"x": 0,
"y": 29
},
"id": 144,
"legend": {
"avg": false,
"current": false,
"max": false,
"min": false,
"show": true,
"total": false,
"values": false
},
"lines": true,
"linewidth": 1,
"links": [],
"nullPointMode": "null",
"percentage": false,
"pointradius": 5,
"points": false,
"renderer": "flot",
"seriesOverrides": [
{
"alias": "/.*_ops_.*/",
"yaxis": 2
}
],
"spaceLength": 10,
"stack": false,
"steppedLine": false,
"targets": [
{
"expr": "[[app]]_dataNode_OpCreateExtent{app=~\"$app\", cluster=\"$cluster\"}",
"format": "time_series",
"hide": false,
"intervalFactor": 1,
"legendFormat": "CreateExtent_{{instance}}",
"refId": "O"
},
{
"expr": "rate([[app]]_dataNode_OpCreateExtent{app=~\"$app\", cluster=\"$cluster\"}[1m])",
"format": "time_series",
"hide": false,
"intervalFactor": 1,
"legendFormat": "CreateExtent_ops_{{instance}}",
"refId": "A"
}
],
"thresholds": [],
"timeFrom": null,
"timeShift": null,
"title": "CreateExtent",
"tooltip": {
"shared": true,
"sort": 0,
"value_type": "individual"
},
"type": "graph",
"xaxis": {
"buckets": null,
"mode": "time",
"name": null,
"show": true,
"values": []
},
"yaxes": [
{
"format": "ns",
"label": null,
"logBase": 1,
"max": null,
"min": null,
"show": true
},
{
"format": "ops",
"label": null,
"logBase": 1,
"max": null,
"min": null,
"show": true
}
],
"yaxis": {
"align": false,
"alignLevel": null
}
},
{
"aliasColors": {},
"bars": false,
"dashLength": 10,
"dashes": false,
"datasource": null,
"fill": 1,
"gridPos": {
"h": 8,
"w": 8,
"x": 8,
"y": 29
},
"id": 145,
"legend": {
"avg": false,
"current": false,
"max": false,
"min": false,
"show": true,
"total": false,
"values": false
},
"lines": true,
"linewidth": 1,
"links": [],
"nullPointMode": "null",
"percentage": false,
"pointradius": 5,
"points": false,
"renderer": "flot",
"seriesOverrides": [
{
"alias": "/.*_ops_.*/",
"yaxis": 2
}
],
"spaceLength": 10,
"stack": false,
"steppedLine": false,
"targets": [
{
"expr": "[[app]]_dataNode_OpExtentRepairRead{app=~\"$app\", cluster=\"$cluster\"}",
"format": "time_series",
"hide": false,
"intervalFactor": 1,
"legendFormat": "ExtentRepairdRead_{{instance}}",
"refId": "P"
},
{
"expr": "rate([[app]]_dataNode_OpExtentRepairRead_count{app=~\"$app\", cluster=\"$cluster\"}[1m])",
"format": "time_series",
"hide": false,
"intervalFactor": 1,
"legendFormat": "ExtentRepairdRead_ops_{{instance}}",
"refId": "A"
}
],
"thresholds": [],
"timeFrom": null,
"timeShift": null,
"title": "ExtentRepairdRead",
"tooltip": {
"shared": true,
"sort": 0,
"value_type": "individual"
},
"type": "graph",
"xaxis": {
"buckets": null,
"mode": "time",
"name": null,
"show": true,
"values": []
},
"yaxes": [
{
"format": "ns",
"label": null,
"logBase": 1,
"max": null,
"min": null,
"show": true
},
{
"format": "short",
"label": null,
"logBase": 1,
"max": null,
"min": null,
"show": true
}
],
"yaxis": {
"align": false,
"alignLevel": null
}
},
{
"aliasColors": {},
"bars": false,
"dashLength": 10,
"dashes": false,
"datasource": null,
"fill": 1,
"gridPos": {
"h": 8,
"w": 8,
"x": 16,
"y": 29
},
"id": 74,
"legend": {
"avg": false,
"current": false,
"max": false,
"min": false,
"show": true,
"total": false,
"values": false
},
"lines": true,
"linewidth": 1,
"links": [],
"nullPointMode": "null",
"percentage": false,
"pointradius": 5,
"points": false,
"renderer": "flot",
"seriesOverrides": [
{
"alias": "/.*_ops_.*/",
"yaxis": 2
}
],
"spaceLength": 10,
"stack": false,
"steppedLine": false,
"targets": [
{
"expr": "[[app]]_dataNode_OpBroadcastMinAppliedID{app=~\"$app\", cluster=\"$cluster\"}",
"format": "time_series",
"intervalFactor": 1,
"legendFormat": "OpBroadcastMinAppliedID_{{instance}}",
"refId": "B"
},
{
"expr": "rate([[app]]_dataNode_OpBroadcastMinAppliedID{app=~\"$app\", cluster=\"$cluster\"}[1m])",
"format": "time_series",
"intervalFactor": 1,
"legendFormat": "OpBroadcastMinAppliedID_ops_{{instance}}",
"refId": "Q"
}
],
"thresholds": [],
"timeFrom": null,
"timeShift": null,
"title": "OpBroadcastMinAppliedID",
"tooltip": {
"shared": true,
"sort": 0,
"value_type": "individual"
},
"type": "graph",
"xaxis": {
"buckets": null,
"mode": "time",
"name": null,
"show": true,
"values": []
},
"yaxes": [
{
"format": "ns",
"label": null,
"logBase": 1,
"max": null,
"min": null,
"show": true
},
{
"format": "ops",
"label": null,
"logBase": 1,
"max": null,
"min": null,
"show": true
}
],
"yaxis": {
"align": false,
"alignLevel": null
}
},
{
"aliasColors": {},
"bars": false,
"dashLength": 10,
"dashes": false,
"datasource": null,
"fill": 1,
"gridPos": {
"h": 8,
"w": 8,
"x": 0,
"y": 37
},
"id": 141,
"legend": {
"avg": false,
"current": false,
"max": false,
"min": false,
"show": true,
"total": false,
"values": false
},
"lines": true,
"linewidth": 1,
"links": [],
"nullPointMode": "null",
"percentage": false,
"pointradius": 5,
"points": false,
"renderer": "flot",
"seriesOverrides": [
{
"alias": "/.*_ops_.*/",
"yaxis": 2
}
],
"spaceLength": 10,
"stack": false,
"steppedLine": false,
"targets": [
{
"expr": "[[app]]_dataNode_OpReadTinyDelete{app=~\"$app\", cluster=\"$cluster\"}",
"format": "time_series",
"hide": false,
"intervalFactor": 1,
"legendFormat": "ReadTinyDelete_{{instance}}",
"refId": "L"
},
{
"expr": "rate([[app]]_dataNode_OpReadTinyDelete_count{app=~\"$app\", cluster=\"$cluster\"}[1m])",
"format": "time_series",
"hide": false,
"intervalFactor": 1,
"legendFormat": "ReadTinyDelete_ops_{{instance}}",
"refId": "A"
}
],
"thresholds": [],
"timeFrom": null,
"timeShift": null,
"title": "ReadTinyDelete",
"tooltip": {
"shared": true,
"sort": 0,
"value_type": "individual"
},
"type": "graph",
"xaxis": {
"buckets": null,
"mode": "time",
"name": null,
"show": true,
"values": []
},
"yaxes": [
{
"format": "ns",
"label": null,
"logBase": 1,
"max": null,
"min": null,
"show": true
},
{
"format": "ops",
"label": null,
"logBase": 1,
"max": null,
"min": null,
"show": true
}
],
"yaxis": {
"align": false,
"alignLevel": null
}
},
{
"aliasColors": {},
"bars": false,
"dashLength": 10,
"dashes": false,
"datasource": null,
"fill": 1,
"gridPos": {
"h": 8,
"w": 8,
"x": 8,
"y": 37
},
"id": 136,
"legend": {
"avg": false,
"current": false,
"max": false,
"min": false,
"show": true,
"total": false,
"values": false
},
"lines": true,
"linewidth": 1,
"links": [],
"nullPointMode": "null",
"percentage": false,
"pointradius": 5,
"points": false,
"renderer": "flot",
"seriesOverrides": [],
"spaceLength": 10,
"stack": false,
"steppedLine": false,
"targets": [
{
"expr": "[[app]]_dataNode_OpBroadcastMinAppliedID{app=~\"$app\", cluster=\"$cluster\"}",
"format": "time_series",
"hide": true,
"intervalFactor": 1,
"legendFormat": "OpBroadcastMinAppliedID_{{instance}}",
"refId": "B"
},
{
"expr": "[[app]]_dataNode_OpCreateDataPartition{app=~\"$app\", cluster=\"$cluster\"}",
"format": "time_series",
"hide": true,
"intervalFactor": 1,
"legendFormat": "CreateDataPartition_{{instance}}",
"refId": "C"
},
{
"expr": "[[app]]_dataNode_OpDataNodeHeartbeat{app=~\"$app\", cluster=\"$cluster\"}",
"format": "time_series",
"hide": true,
"intervalFactor": 1,
"legendFormat": "DataNodeHeartbeat_{{instance}}",
"refId": "D"
},
{
"expr": "[[app]]_dataNode_OpDecommissionDataPartition{app=~\"$app\", cluster=\"$cluster\"}",
"format": "time_series",
"hide": true,
"intervalFactor": 1,
"legendFormat": "DecommissionDataPartition_{{instance}}",
"refId": "A"
},
{
"expr": "[[app]]_dataNode_OpGetAllWatermarks{app=~\"$app\", cluster=\"$cluster\"}",
"format": "time_series",
"hide": true,
"intervalFactor": 1,
"legendFormat": "GetAllWatermarks_{{instance}}",
"refId": "E"
},
{
"expr": "[[app]]_dataNode_OpGetAppliedId{app=~\"$app\", cluster=\"$cluster\"}",
"format": "time_series",
"hide": true,
"intervalFactor": 1,
"legendFormat": "GetAppliedId_{{instance}}",
"refId": "F"
},
{
"expr": "[[app]]_dataNode_OpGetPartitionSize{app=~\"$app\", cluster=\"$cluster\"}",
"format": "time_series",
"hide": false,
"intervalFactor": 1,
"legendFormat": "GetPartitionSize_{{instance}}",
"refId": "G"
},
{
"expr": "[[app]]_dataNode_OpLoadDataPartition{app=~\"$app\", cluster=\"$cluster\"}",
"format": "time_series",
"hide": true,
"intervalFactor": 1,
"legendFormat": "LoadDataPartition_{{instance}}",
"refId": "H"
},
{
"expr": "[[app]]_dataNode_OpMarkDelete{app=~\"$app\", cluster=\"$cluster\"}",
"format": "time_series",
"hide": true,
"intervalFactor": 1,
"legendFormat": "MarkDelete_{{instance}}",
"refId": "I"
},
{
"expr": "[[app]]_dataNode_OpNotifyReplicasToRepair{app=~\"$app\", cluster=\"$cluster\"}",
"format": "time_series",
"hide": true,
"intervalFactor": 1,
"legendFormat": "NotifyReplicasToRepair_{{instance}}",
"refId": "J"
},
{
"expr": "[[app]]_dataNode_OpRandomWrite{app=~\"$app\", cluster=\"$cluster\"}",
"format": "time_series",
"hide": true,
"intervalFactor": 1,
"legendFormat": "OpRandomWrite_{{instance}}",
"refId": "K"
},
{
"expr": "[[app]]_dataNode_OpReadTinyDelete{app=~\"$app\", cluster=\"$cluster\"}",
"format": "time_series",
"hide": true,
"intervalFactor": 1,
"legendFormat": "ReadTinyDelete_{{instance}}",
"refId": "L"
},
{
"expr": "[[app]]_dataNode_OpStreamRead{app=~\"$app\", cluster=\"$cluster\"}",
"format": "time_series",
"hide": true,
"intervalFactor": 1,
"legendFormat": "StreamRead_{{instance}}",
"refId": "M"
},
{
"expr": "[[app]]_dataNode_OpWrite{app=~\"$app\", cluster=\"$cluster\"}",
"format": "time_series",
"hide": true,
"intervalFactor": 1,
"legendFormat": "Write_{{instance}}",
"refId": "N"
},
{
"expr": "[[app]]_dataNode_OpCreateExtent{app=~\"$app\", cluster=\"$cluster\"}",
"format": "time_series",
"hide": true,
"intervalFactor": 1,
"legendFormat": "CreateExtent_{{instance}}",
"refId": "O"
},
{
"expr": "[[app]]_dataNode_OpExtentRepairRead{app=~\"$app\", cluster=\"$cluster\"}",
"format": "time_series",
"hide": true,
"intervalFactor": 1,
"legendFormat": "ExtentRepairdRead_{{instance}}",
"refId": "P"
}
],
"thresholds": [],
"timeFrom": null,
"timeShift": null,
"title": "GetPartitionSize",
"tooltip": {
"shared": true,
"sort": 0,
"value_type": "individual"
},
"type": "graph",
"xaxis": {
"buckets": null,
"mode": "time",
"name": null,
"show": true,
"values": []
},
"yaxes": [
{
"format": "ns",
"label": null,
"logBase": 1,
"max": null,
"min": null,
"show": true
},
{
"format": "short",
"label": null,
"logBase": 1,
"max": null,
"min": null,
"show": true
}
],
"yaxis": {
"align": false,
"alignLevel": null
}
},
{
"aliasColors": {},
"bars": false,
"dashLength": 10,
"dashes": false,
"datasource": null,
"fill": 1,
"gridPos": {
"h": 8,
"w": 8,
"x": 16,
"y": 37
},
"id": 146,
"legend": {
"avg": false,
"current": false,
"max": false,
"min": false,
"show": true,
"total": false,
"values": false
},
"lines": true,
"linewidth": 1,
"links": [],
"nullPointMode": "null",
"percentage": false,
"pointradius": 5,
"points": false,
"renderer": "flot",
"seriesOverrides": [
{
"alias": "/.*_ops_.*/",
"yaxis": 2
}
],
"spaceLength": 10,
"stack": false,
"steppedLine": false,
"targets": [
{
"expr": "[[app]]_dataNode_OpTinyExtentRepairRead{app=~\"$app\", cluster=\"$cluster\"}",
"format": "time_series",
"hide": false,
"intervalFactor": 1,
"legendFormat": "TinyExtentRepairdRead_{{instance}}",
"refId": "P"
},
{
"expr": "rate([[app]]_dataNode_OpTinyExtentRepairRead_count{app=~\"$app\", cluster=\"$cluster\"}[1m])",
"format": "time_series",
"hide": false,
"intervalFactor": 1,
"legendFormat": "TinyExtentRepairdRead_ops_{{instance}}",
"refId": "A"
}
],
"thresholds": [],
"timeFrom": null,
"timeShift": null,
"title": "TinyExtentRepairdRead",
"tooltip": {
"shared": true,
"sort": 0,
"value_type": "individual"
},
"type": "graph",
"xaxis": {
"buckets": null,
"mode": "time",
"name": null,
"show": true,
"values": []
},
"yaxes": [
{
"format": "ns",
"label": null,
"logBase": 1,
"max": null,
"min": null,
"show": true
},
{
"format": "ops",
"label": null,
"logBase": 1,
"max": null,
"min": null,
"show": true
}
],
"yaxis": {
"align": false,
"alignLevel": null
}
},
{
"aliasColors": {},
"bars": false,
"dashLength": 10,
"dashes": false,
"datasource": null,
"fill": 1,
"gridPos": {
"h": 8,
"w": 8,
"x": 0,
"y": 45
},
"id": 131,
"legend": {
"avg": false,
"current": false,
"max": false,
"min": false,
"show": true,
"total": false,
"values": false
},
"lines": true,
"linewidth": 1,
"links": [],
"nullPointMode": "null",
"percentage": false,
"pointradius": 5,
"points": false,
"renderer": "flot",
"seriesOverrides": [],
"spaceLength": 10,
"stack": false,
"steppedLine": false,
"targets": [
{
"expr": "[[app]]_dataNode_OpBroadcastMinAppliedID{app=~\"$app\", cluster=\"$cluster\"}",
"format": "time_series",
"hide": true,
"intervalFactor": 1,
"legendFormat": "OpBroadcastMinAppliedID_{{instance}}",
"refId": "B"
},
{
"expr": "[[app]]_dataNode_OpCreateDataPartition{app=~\"$app\", cluster=\"$cluster\"}",
"format": "time_series",
"hide": true,
"intervalFactor": 1,
"legendFormat": "CreateDataPartition_{{instance}}",
"refId": "C"
},
{
"expr": "[[app]]_dataNode_OpDataNodeHeartbeat{app=~\"$app\", cluster=\"$cluster\"}",
"format": "time_series",
"hide": true,
"intervalFactor": 1,
"legendFormat": "DataNodeHeartbeat_{{instance}}",
"refId": "D"
},
{
"expr": "[[app]]_dataNode_OpDecommissionDataPartition{app=~\"$app\", cluster=\"$cluster\"}",
"format": "time_series",
"hide": false,
"intervalFactor": 1,
"legendFormat": "DecommissionDataPartition_{{instance}}",
"refId": "A"
},
{
"expr": "[[app]]_dataNode_OpGetAllWatermarks{app=~\"$app\", cluster=\"$cluster\"}",
"format": "time_series",
"hide": true,
"intervalFactor": 1,
"legendFormat": "GetAllWatermarks_{{instance}}",
"refId": "E"
},
{
"expr": "[[app]]_dataNode_OpGetAppliedId{app=~\"$app\", cluster=\"$cluster\"}",
"format": "time_series",
"hide": true,
"intervalFactor": 1,
"legendFormat": "GetAppliedId_{{instance}}",
"refId": "F"
},
{
"expr": "[[app]]_dataNode_OpGetPartitionSize{app=~\"$app\", cluster=\"$cluster\"}",
"format": "time_series",
"hide": true,
"intervalFactor": 1,
"legendFormat": "GetPartitionSize_{{instance}}",
"refId": "G"
},
{
"expr": "[[app]]_dataNode_OpLoadDataPartition{app=~\"$app\", cluster=\"$cluster\"}",
"format": "time_series",
"hide": true,
"intervalFactor": 1,
"legendFormat": "LoadDataPartition_{{instance}}",
"refId": "H"
},
{
"expr": "[[app]]_dataNode_OpMarkDelete{app=~\"$app\", cluster=\"$cluster\"}",
"format": "time_series",
"hide": true,
"intervalFactor": 1,
"legendFormat": "MarkDelete_{{instance}}",
"refId": "I"
},
{
"expr": "[[app]]_dataNode_OpNotifyReplicasToRepair{app=~\"$app\", cluster=\"$cluster\"}",
"format": "time_series",
"hide": true,
"intervalFactor": 1,
"legendFormat": "NotifyReplicasToRepair_{{instance}}",
"refId": "J"
},
{
"expr": "[[app]]_dataNode_OpRandomWrite{app=~\"$app\", cluster=\"$cluster\"}",
"format": "time_series",
"hide": true,
"intervalFactor": 1,
"legendFormat": "OpRandomWrite_{{instance}}",
"refId": "K"
},
{
"expr": "[[app]]_dataNode_OpReadTinyDelete{app=~\"$app\", cluster=\"$cluster\"}",
"format": "time_series",
"hide": true,
"intervalFactor": 1,
"legendFormat": "ReadTinyDelete_{{instance}}",
"refId": "L"
},
{
"expr": "[[app]]_dataNode_OpStreamRead{app=~\"$app\", cluster=\"$cluster\"}",
"format": "time_series",
"hide": true,
"intervalFactor": 1,
"legendFormat": "StreamRead_{{instance}}",
"refId": "M"
},
{
"expr": "[[app]]_dataNode_OpWrite{app=~\"$app\", cluster=\"$cluster\"}",
"format": "time_series",
"hide": true,
"intervalFactor": 1,
"legendFormat": "Write_{{instance}}",
"refId": "N"
},
{
"expr": "[[app]]_dataNode_OpCreateExtent{app=~\"$app\", cluster=\"$cluster\"}",
"format": "time_series",
"hide": true,
"intervalFactor": 1,
"legendFormat": "CreateExtent_{{instance}}",
"refId": "O"
},
{
"expr": "[[app]]_dataNode_OpExtentRepairRead{app=~\"$app\", cluster=\"$cluster\"}",
"format": "time_series",
"hide": true,
"intervalFactor": 1,
"legendFormat": "ExtentRepairdRead_{{instance}}",
"refId": "P"
}
],
"thresholds": [],
"timeFrom": null,
"timeShift": null,
"title": "DecommissionDataPartition",
"tooltip": {
"shared": true,
"sort": 0,
"value_type": "individual"
},
"type": "graph",
"xaxis": {
"buckets": null,
"mode": "time",
"name": null,
"show": true,
"values": []
},
"yaxes": [
{
"format": "ns",
"label": null,
"logBase": 1,
"max": null,
"min": null,
"show": true
},
{
"format": "short",
"label": null,
"logBase": 1,
"max": null,
"min": null,
"show": true
}
],
"yaxis": {
"align": false,
"alignLevel": null
}
},
{
"aliasColors": {},
"bars": false,
"dashLength": 10,
"dashes": false,
"datasource": null,
"fill": 1,
"gridPos": {
"h": 8,
"w": 8,
"x": 8,
"y": 45
},
"id": 140,
"legend": {
"avg": false,
"current": false,
"max": false,
"min": false,
"show": true,
"total": false,
"values": false
},
"lines": true,
"linewidth": 1,
"links": [],
"nullPointMode": "null",
"percentage": false,
"pointradius": 5,
"points": false,
"renderer": "flot",
"seriesOverrides": [],
"spaceLength": 10,
"stack": false,
"steppedLine": false,
"targets": [
{
"expr": "[[app]]_dataNode_OpBroadcastMinAppliedID{app=~\"$app\", cluster=\"$cluster\"}",
"format": "time_series",
"hide": true,
"intervalFactor": 1,
"legendFormat": "OpBroadcastMinAppliedID_{{instance}}",
"refId": "B"
},
{
"expr": "[[app]]_dataNode_OpCreateDataPartition{app=~\"$app\", cluster=\"$cluster\"}",
"format": "time_series",
"hide": true,
"intervalFactor": 1,
"legendFormat": "CreateDataPartition_{{instance}}",
"refId": "C"
},
{
"expr": "[[app]]_dataNode_OpDataNodeHeartbeat{app=~\"$app\", cluster=\"$cluster\"}",
"format": "time_series",
"hide": true,
"intervalFactor": 1,
"legendFormat": "DataNodeHeartbeat_{{instance}}",
"refId": "D"
},
{
"expr": "[[app]]_dataNode_OpDecommissionDataPartition{app=~\"$app\", cluster=\"$cluster\"}",
"format": "time_series",
"hide": true,
"intervalFactor": 1,
"legendFormat": "DecommissionDataPartition_{{instance}}",
"refId": "A"
},
{
"expr": "[[app]]_dataNode_OpGetAllWatermarks{app=~\"$app\", cluster=\"$cluster\"}",
"format": "time_series",
"hide": true,
"intervalFactor": 1,
"legendFormat": "GetAllWatermarks_{{instance}}",
"refId": "E"
},
{
"expr": "[[app]]_dataNode_OpGetAppliedId{app=~\"$app\", cluster=\"$cluster\"}",
"format": "time_series",
"hide": true,
"intervalFactor": 1,
"legendFormat": "GetAppliedId_{{instance}}",
"refId": "F"
},
{
"expr": "[[app]]_dataNode_OpGetPartitionSize{app=~\"$app\", cluster=\"$cluster\"}",
"format": "time_series",
"hide": true,
"intervalFactor": 1,
"legendFormat": "GetPartitionSize_{{instance}}",
"refId": "G"
},
{
"expr": "[[app]]_dataNode_OpLoadDataPartition{app=~\"$app\", cluster=\"$cluster\"}",
"format": "time_series",
"hide": true,
"intervalFactor": 1,
"legendFormat": "LoadDataPartition_{{instance}}",
"refId": "H"
},
{
"expr": "[[app]]_dataNode_OpMarkDelete{app=~\"$app\", cluster=\"$cluster\"}",
"format": "time_series",
"hide": true,
"intervalFactor": 1,
"legendFormat": "MarkDelete_{{instance}}",
"refId": "I"
},
{
"expr": "[[app]]_dataNode_OpNotifyReplicasToRepair{app=~\"$app\", cluster=\"$cluster\"}",
"format": "time_series",
"hide": true,
"intervalFactor": 1,
"legendFormat": "NotifyReplicasToRepair_{{instance}}",
"refId": "J"
},
{
"expr": "[[app]]_dataNode_OpRandomWrite{app=~\"$app\", cluster=\"$cluster\"}",
"format": "time_series",
"hide": false,
"intervalFactor": 1,
"legendFormat": "OpRandomWrite_{{instance}}",
"refId": "K"
},
{
"expr": "[[app]]_dataNode_OpReadTinyDelete{app=~\"$app\", cluster=\"$cluster\"}",
"format": "time_series",
"hide": true,
"intervalFactor": 1,
"legendFormat": "ReadTinyDelete_{{instance}}",
"refId": "L"
},
{
"expr": "[[app]]_dataNode_OpStreamRead{app=~\"$app\", cluster=\"$cluster\"}",
"format": "time_series",
"hide": true,
"intervalFactor": 1,
"legendFormat": "StreamRead_{{instance}}",
"refId": "M"
},
{
"expr": "[[app]]_dataNode_OpWrite{app=~\"$app\", cluster=\"$cluster\"}",
"format": "time_series",
"hide": true,
"intervalFactor": 1,
"legendFormat": "Write_{{instance}}",
"refId": "N"
},
{
"expr": "[[app]]_dataNode_OpCreateExtent{app=~\"$app\", cluster=\"$cluster\"}",
"format": "time_series",
"hide": true,
"intervalFactor": 1,
"legendFormat": "CreateExtent_{{instance}}",
"refId": "O"
},
{
"expr": "[[app]]_dataNode_OpExtentRepairRead{app=~\"$app\", cluster=\"$cluster\"}",
"format": "time_series",
"hide": true,
"intervalFactor": 1,
"legendFormat": "ExtentRepairdRead_{{instance}}",
"refId": "P"
}
],
"thresholds": [],
"timeFrom": null,
"timeShift": null,
"title": "OpRandomWrite",
"tooltip": {
"shared": true,
"sort": 0,
"value_type": "individual"
},
"type": "graph",
"xaxis": {
"buckets": null,
"mode": "time",
"name": null,
"show": true,
"values": []
},
"yaxes": [
{
"format": "ns",
"label": null,
"logBase": 1,
"max": null,
"min": null,
"show": true
},
{
"format": "short",
"label": null,
"logBase": 1,
"max": null,
"min": null,
"show": true
}
],
"yaxis": {
"align": false,
"alignLevel": null
}
}
],
"title": "Datanode",
"type": "row"
},
{
"collapsed": true,
"gridPos": {
"h": 1,
"w": 24,
"x": 0,
"y": 38
},
"id": 66,
"panels": [
{
"aliasColors": {},
"bars": false,
"dashLength": 10,
"dashes": false,
"datasource": null,
"fill": 1,
"gridPos": {
"h": 7,
"w": 8,
"x": 0,
"y": 6
},
"id": 64,
"legend": {
"avg": false,
"current": false,
"max": false,
"min": false,
"show": true,
"total": false,
"values": false
},
"lines": true,
"linewidth": 1,
"links": [],
"nullPointMode": "null",
"percentage": false,
"pointradius": 5,
"points": false,
"renderer": "flot",
"seriesOverrides": [
{
"alias": "/.*_ops_.*/",
"yaxis": 2
}
],
"spaceLength": 10,
"stack": false,
"steppedLine": false,
"targets": [
{
"expr": "[[app]]_fuseclient_OpMetaReadDir{app=\"$app\", cluster=\"$cluster\", instance=\"$instance\"}",
"format": "time_series",
"hide": false,
"intervalFactor": 1,
"legendFormat": "MetaReaddir_{{instance}}",
"refId": "K"
},
{
"expr": "rate([[app]]_fuseclient_OpMetaReadDir_count{app=\"$app\", cluster=\"$cluster\", instance=\"$instance\"}[1m])",
"format": "time_series",
"hide": false,
"intervalFactor": 1,
"legendFormat": "MetaReaddir_ops_{{instance}}",
"refId": "E"
}
],
"thresholds": [],
"timeFrom": null,
"timeShift": null,
"title": "MetaReaddir",
"tooltip": {
"shared": true,
"sort": 0,
"value_type": "individual"
},
"type": "graph",
"xaxis": {
"buckets": null,
"mode": "time",
"name": null,
"show": true,
"values": []
},
"yaxes": [
{
"format": "ns",
"label": null,
"logBase": 1,
"max": null,
"min": null,
"show": true
},
{
"format": "ops",
"label": null,
"logBase": 1,
"max": null,
"min": null,
"show": true
}
],
"yaxis": {
"align": false,
"alignLevel": null
}
},
{
"aliasColors": {},
"bars": false,
"dashLength": 10,
"dashes": false,
"datasource": null,
"fill": 1,
"gridPos": {
"h": 7,
"w": 8,
"x": 8,
"y": 6
},
"id": 157,
"legend": {
"avg": false,
"current": false,
"max": false,
"min": false,
"show": true,
"total": false,
"values": false
},
"lines": true,
"linewidth": 1,
"links": [],
"nullPointMode": "null",
"percentage": false,
"pointradius": 5,
"points": false,
"renderer": "flot",
"seriesOverrides": [
{
"alias": "/.*_count_.*/",
"yaxis": 2
}
],
"spaceLength": 10,
"stack": false,
"steppedLine": false,
"targets": [
{
"expr": "[[app]]_fuseclient_OpMetaBatchInodeGet{app=~\"$app\", cluster=\"$cluster\"}",
"format": "time_series",
"hide": true,
"intervalFactor": 1,
"legendFormat": "OpMetaBatchInodeGet_{{instance}}",
"refId": "A"
},
{
"expr": "[[app]]_fuseclient_OpMetaCreateInode{app=~\"$app\", cluster=\"$cluster\", instance=\"$instance\"}",
"format": "time_series",
"hide": false,
"intervalFactor": 1,
"legendFormat": "CreateInode_{{instance}}",
"refId": "D"
},
{
"expr": "[[app]]_fuseclient_OpMetaDeleteInode{app=~\"$app\", cluster=\"$cluster\"}",
"format": "time_series",
"hide": true,
"intervalFactor": 1,
"legendFormat": "DeleteInode_{{instance}}",
"refId": "F"
},
{
"expr": "[[app]]_fuseclient_OpMetaEvictInode{app=~\"$app\", cluster=\"$cluster\"}",
"format": "time_series",
"hide": true,
"intervalFactor": 1,
"legendFormat": "EvictInode_{{instance}}",
"refId": "G"
},
{
"expr": "[[app]]_fuseclient_OpMetaInodeGet{app=~\"$app\", cluster=\"$cluster\"}",
"format": "time_series",
"hide": true,
"intervalFactor": 1,
"legendFormat": "InodeGet_{{instance}}",
"refId": "J"
},
{
"expr": "[[app]]_fuseclient_OpMetaTruncate{app=~\"$app\", cluster=\"$cluster\"}",
"format": "time_series",
"hide": true,
"intervalFactor": 1,
"legendFormat": "Truncate_{{instance}}",
"refId": "B"
},
{
"expr": "rate([[app]]_fuseclient_OpMetaBatchInodeGet_count{app=~\"$app\", cluster=\"$cluster\"}[1m])",
"format": "time_series",
"hide": true,
"intervalFactor": 1,
"legendFormat": "OpMetaBatchInodeGet_count_{{instance}}",
"refId": "C"
},
{
"expr": "rate([[app]]_fuseclient_OpMetaCreateInode{app=~\"$app\", cluster=\"$cluster\", instance=\"$instance\"}[1m])",
"format": "time_series",
"hide": false,
"intervalFactor": 1,
"legendFormat": "CreateInode_count_{{instance}}",
"refId": "E"
},
{
"expr": "rate([[app]]_fuseclient_OpMetaDeleteInode{app=~\"$app\", cluster=\"$cluster\"}[1m])",
"format": "time_series",
"hide": true,
"intervalFactor": 1,
"legendFormat": "DeleteInode_count_{{instance}}",
"refId": "H"
},
{
"expr": "rate([[app]]_fuseclient_OpMetaEvictInode{app=~\"$app\", cluster=\"$cluster\"}[1m])",
"format": "time_series",
"hide": true,
"intervalFactor": 1,
"legendFormat": "EvictInode_count_{{instance}}",
"refId": "I"
},
{
"expr": "rate([[app]]_fuseclient_OpMetaInodeGet{app=~\"$app\", cluster=\"$cluster\"}[1m])",
"format": "time_series",
"hide": true,
"intervalFactor": 1,
"legendFormat": "InodeGet_count_{{instance}}",
"refId": "K"
},
{
"expr": "rate([[app]]_fuseclient_OpMetaTruncate{app=~\"$app\", cluster=\"$cluster\"}[1m])",
"format": "time_series",
"hide": true,
"intervalFactor": 1,
"legendFormat": "Truncate_count_{{instance}}",
"refId": "L"
}
],
"thresholds": [],
"timeFrom": null,
"timeShift": null,
"title": "OpMetaCreateInode",
"tooltip": {
"shared": true,
"sort": 0,
"value_type": "individual"
},
"type": "graph",
"xaxis": {
"buckets": null,
"mode": "time",
"name": null,
"show": true,
"values": []
},
"yaxes": [
{
"format": "ns",
"label": null,
"logBase": 1,
"max": null,
"min": null,
"show": true
},
{
"format": "ops",
"label": null,
"logBase": 1,
"max": null,
"min": null,
"show": true
}
],
"yaxis": {
"align": false,
"alignLevel": null
}
},
{
"aliasColors": {},
"bars": false,
"dashLength": 10,
"dashes": false,
"datasource": null,
"fill": 1,
"gridPos": {
"h": 7,
"w": 8,
"x": 16,
"y": 6
},
"id": 156,
"legend": {
"avg": false,
"current": false,
"max": false,
"min": false,
"show": true,
"total": false,
"values": false
},
"lines": true,
"linewidth": 1,
"links": [],
"nullPointMode": "null",
"percentage": false,
"pointradius": 5,
"points": false,
"renderer": "flot",
"seriesOverrides": [
{
"alias": "/.*_count_.*/",
"yaxis": 2
}
],
"spaceLength": 10,
"stack": false,
"steppedLine": false,
"targets": [
{
"expr": "[[app]]_fuseclient_OpMetaBatchInodeGet{app=~\"$app\", cluster=\"$cluster\"}",
"format": "time_series",
"hide": true,
"intervalFactor": 1,
"legendFormat": "OpMetaBatchInodeGet_{{instance}}",
"refId": "A"
},
{
"expr": "[[app]]_fuseclient_OpMetaCreateInode{app=~\"$app\", cluster=\"$cluster\"}",
"format": "time_series",
"hide": true,
"intervalFactor": 1,
"legendFormat": "CreateInode_{{instance}}",
"refId": "D"
},
{
"expr": "[[app]]_fuseclient_OpMetaDeleteInode{app=~\"$app\", cluster=\"$cluster\", instance=\"$instance\"}",
"format": "time_series",
"hide": false,
"intervalFactor": 1,
"legendFormat": "DeleteInode_{{instance}}",
"refId": "F"
},
{
"expr": "[[app]]_fuseclient_OpMetaEvictInode{app=~\"$app\", cluster=\"$cluster\"}",
"format": "time_series",
"hide": true,
"intervalFactor": 1,
"legendFormat": "EvictInode_{{instance}}",
"refId": "G"
},
{
"expr": "[[app]]_fuseclient_OpMetaInodeGet{app=~\"$app\", cluster=\"$cluster\"}",
"format": "time_series",
"hide": true,
"intervalFactor": 1,
"legendFormat": "InodeGet_{{instance}}",
"refId": "J"
},
{
"expr": "[[app]]_fuseclient_OpMetaTruncate{app=~\"$app\", cluster=\"$cluster\"}",
"format": "time_series",
"hide": true,
"intervalFactor": 1,
"legendFormat": "Truncate_{{instance}}",
"refId": "B"
},
{
"expr": "rate([[app]]_fuseclient_OpMetaBatchInodeGet_count{app=~\"$app\", cluster=\"$cluster\", instance=\"$instance\"}[1m])",
"format": "time_series",
"hide": true,
"intervalFactor": 1,
"legendFormat": "OpMetaBatchInodeGet_count_{{instance}}",
"refId": "C"
},
{
"expr": "rate([[app]]_fuseclient_OpMetaCreateInode{app=~\"$app\", cluster=\"$cluster\"}[1m])",
"format": "time_series",
"hide": true,
"intervalFactor": 1,
"legendFormat": "CreateInode_count_{{instance}}",
"refId": "E"
},
{
"expr": "rate([[app]]_fuseclient_OpMetaDeleteInode_count{app=~\"$app\", cluster=\"$cluster\", instance=\"$instance\"}[1m])",
"format": "time_series",
"hide": false,
"intervalFactor": 1,
"legendFormat": "DeleteInode_count_{{instance}}",
"refId": "H"
},
{
"expr": "rate([[app]]_fuseclient_OpMetaEvictInode{app=~\"$app\", cluster=\"$cluster\"}[1m])",
"format": "time_series",
"hide": true,
"intervalFactor": 1,
"legendFormat": "EvictInode_count_{{instance}}",
"refId": "I"
},
{
"expr": "rate([[app]]_fuseclient_OpMetaInodeGet{app=~\"$app\", cluster=\"$cluster\"}[1m])",
"format": "time_series",
"hide": true,
"intervalFactor": 1,
"legendFormat": "InodeGet_count_{{instance}}",
"refId": "K"
},
{
"expr": "rate([[app]]_fuseclient_OpMetaTruncate{app=~\"$app\", cluster=\"$cluster\"}[1m])",
"format": "time_series",
"hide": true,
"intervalFactor": 1,
"legendFormat": "Truncate_count_{{instance}}",
"refId": "L"
}
],
"thresholds": [],
"timeFrom": null,
"timeShift": null,
"title": "DeleteInode",
"tooltip": {
"shared": true,
"sort": 0,
"value_type": "individual"
},
"type": "graph",
"xaxis": {
"buckets": null,
"mode": "time",
"name": null,
"show": true,
"values": []
},
"yaxes": [
{
"format": "ns",
"label": null,
"logBase": 1,
"max": null,
"min": null,
"show": true
},
{
"format": "ops",
"label": null,
"logBase": 1,
"max": null,
"min": null,
"show": true
}
],
"yaxis": {
"align": false,
"alignLevel": null
}
},
{
"aliasColors": {},
"bars": false,
"dashLength": 10,
"dashes": false,
"datasource": null,
"fill": 1,
"gridPos": {
"h": 7,
"w": 8,
"x": 0,
"y": 13
},
"id": 160,
"legend": {
"avg": false,
"current": false,
"max": false,
"min": false,
"show": true,
"total": false,
"values": false
},
"lines": true,
"linewidth": 1,
"links": [],
"nullPointMode": "null",
"percentage": false,
"pointradius": 5,
"points": false,
"renderer": "flot",
"seriesOverrides": [
{
"alias": "/.*_count_.*/",
"yaxis": 2
}
],
"spaceLength": 10,
"stack": false,
"steppedLine": false,
"targets": [
{
"expr": "[[app]]_fuseclient_OpMetaCreateDentry{app=~\"$app\", cluster=\"$cluster\"}",
"format": "time_series",
"hide": true,
"intervalFactor": 1,
"legendFormat": "OpMetaCreateDentry_{{instance}}",
"refId": "C"
},
{
"expr": "[[app]]_fuseclient_OpMetaDeleteDentry{app=~\"$app\", cluster=\"$cluster\", instance=\"$instance\"}",
"format": "time_series",
"hide": false,
"intervalFactor": 1,
"legendFormat": "OpMetaDeleteDentry_{{instance}}",
"refId": "E"
},
{
"expr": "[[app]]_fuseclient_OpMetaUpdateDentry{app=~\"$app\", cluster=\"$cluster\"}",
"format": "time_series",
"hide": true,
"intervalFactor": 1,
"legendFormat": "OpMetaUpdateDentry_{{instance}}",
"refId": "A"
},
{
"expr": "[[app]]_fuseclient_OpMetaCreateDentry_count{app=~\"$app\", cluster=\"$cluster\"}",
"format": "time_series",
"hide": true,
"intervalFactor": 1,
"legendFormat": "OpMetaCreateDentry_count_{{instance}}",
"refId": "B"
},
{
"expr": "rate([[app]]_fuseclient_OpMetaDeleteDentry_count{app=~\"$app\", cluster=\"$cluster\", instance=\"$instance\"}[1m])",
"format": "time_series",
"hide": false,
"intervalFactor": 1,
"legendFormat": "OpMetaDeleteDentry_count_{{instance}}",
"refId": "D"
},
{
"expr": "[[app]]_fuseclient_OpMetaUpdateDentry_count{app=~\"$app\", cluster=\"$cluster\"}",
"format": "time_series",
"hide": true,
"intervalFactor": 1,
"legendFormat": "OpMetaUpdateDentry_count_{{instance}}",
"refId": "F"
}
],
"thresholds": [],
"timeFrom": null,
"timeShift": null,
"title": "OpMetaDeleteDentry",
"tooltip": {
"shared": true,
"sort": 0,
"value_type": "individual"
},
"type": "graph",
"xaxis": {
"buckets": null,
"mode": "time",
"name": null,
"show": true,
"values": []
},
"yaxes": [
{
"format": "ns",
"label": null,
"logBase": 1,
"max": null,
"min": null,
"show": true
},
{
"format": "ops",
"label": null,
"logBase": 1,
"max": null,
"min": null,
"show": true
}
],
"yaxis": {
"align": false,
"alignLevel": null
}
},
{
"aliasColors": {},
"bars": false,
"dashLength": 10,
"dashes": false,
"datasource": null,
"fill": 1,
"gridPos": {
"h": 7,
"w": 8,
"x": 8,
"y": 13
},
"id": 68,
"legend": {
"avg": false,
"current": false,
"max": false,
"min": false,
"show": true,
"total": false,
"values": false
},
"lines": true,
"linewidth": 1,
"links": [],
"nullPointMode": "null",
"percentage": false,
"pointradius": 5,
"points": false,
"renderer": "flot",
"seriesOverrides": [
{
"alias": "/.*_count_.*/",
"yaxis": 2
}
],
"spaceLength": 10,
"stack": false,
"steppedLine": false,
"targets": [
{
"expr": "[[app]]_fuseclient_OpMetaCreateDentry{app=~\"$app\", cluster=\"$cluster\", instance=\"$instance\"}",
"format": "time_series",
"intervalFactor": 1,
"legendFormat": "OpMetaCreateDentry_{{instance}}",
"refId": "C"
},
{
"expr": "[[app]]_fuseclient_OpMetaDeleteDentry{app=~\"$app\", cluster=\"$cluster\"}",
"format": "time_series",
"hide": true,
"intervalFactor": 1,
"legendFormat": "OpMetaDeleteDentry_{{instance}}",
"refId": "E"
},
{
"expr": "[[app]]_fuseclient_OpMetaUpdateDentry{app=~\"$app\", cluster=\"$cluster\"}",
"format": "time_series",
"hide": true,
"intervalFactor": 1,
"legendFormat": "OpMetaUpdateDentry_{{instance}}",
"refId": "A"
},
{
"expr": "rate([[app]]_fuseclient_OpMetaCreateDentry_count{app=~\"$app\", cluster=\"$cluster\", instance=\"$instance\"}[1m])",
"format": "time_series",
"hide": false,
"intervalFactor": 1,
"legendFormat": "OpMetaCreateDentry_count_{{instance}}",
"refId": "B"
},
{
"expr": "[[app]]_fuseclient_OpMetaDeleteDentry_count{app=~\"$app\", cluster=\"$cluster\"}",
"format": "time_series",
"hide": true,
"intervalFactor": 1,
"legendFormat": "OpMetaDeleteDentry_count_{{instance}}",
"refId": "D"
},
{
"expr": "[[app]]_fuseclient_OpMetaUpdateDentry_count{app=~\"$app\", cluster=\"$cluster\"}",
"format": "time_series",
"hide": true,
"intervalFactor": 1,
"legendFormat": "OpMetaUpdateDentry_count_{{instance}}",
"refId": "F"
}
],
"thresholds": [],
"timeFrom": null,
"timeShift": null,
"title": "OpMetaCreateDentry",
"tooltip": {
"shared": true,
"sort": 0,
"value_type": "individual"
},
"type": "graph",
"xaxis": {
"buckets": null,
"mode": "time",
"name": null,
"show": true,
"values": []
},
"yaxes": [
{
"format": "ns",
"label": null,
"logBase": 1,
"max": null,
"min": null,
"show": true
},
{
"format": "ops",
"label": null,
"logBase": 1,
"max": null,
"min": null,
"show": true
}
],
"yaxis": {
"align": false,
"alignLevel": null
}
},
{
"aliasColors": {},
"bars": false,
"dashLength": 10,
"dashes": false,
"datasource": null,
"fill": 1,
"gridPos": {
"h": 7,
"w": 8,
"x": 16,
"y": 13
},
"id": 102,
"legend": {
"avg": false,
"current": false,
"max": false,
"min": false,
"show": true,
"total": false,
"values": false
},
"lines": true,
"linewidth": 1,
"links": [],
"nullPointMode": "null",
"percentage": false,
"pointradius": 5,
"points": false,
"renderer": "flot",
"seriesOverrides": [
{
"alias": "/.*_count_.*/",
"yaxis": 2
}
],
"spaceLength": 10,
"stack": false,
"steppedLine": false,
"targets": [
{
"expr": "rate([[app]]_fuseclient_StreamWrite_count{app=\"$app\",cluster=\"$cluster\"}[1m])",
"format": "time_series",
"hide": true,
"intervalFactor": 1,
"legendFormat": "StreamWrite_{{instance}}",
"refId": "A"
},
{
"expr": "rate([[app]]_fuseclient_StreamRead_count{app=\"$app\",cluster=\"$cluster\"}[1m])",
"format": "time_series",
"hide": true,
"intervalFactor": 1,
"legendFormat": "StreamRead",
"refId": "B"
},
{
"expr": "rate([[app]]_fuseclient_StreamSyncWrite_count{instance=\"$instance\",cluster=\"$cluster\"}[1m])",
"format": "time_series",
"hide": true,
"intervalFactor": 1,
"legendFormat": "SyncWrite",
"refId": "C"
},
{
"expr": "rate([[app]]_fuseclient_StreamSyncOverwrite_count{instance=\"$instance\",cluster=\"$cluster\"}[1m])",
"format": "time_series",
"hide": true,
"intervalFactor": 1,
"legendFormat": "SyncOverwrite",
"refId": "D"
},
{
"expr": "rate([[app]]_fuseclient_StreamOverwrite_count{instance=\"$instance\",cluster=\"$cluster\"}[1m])",
"format": "time_series",
"hide": true,
"intervalFactor": 1,
"legendFormat": "Overwrite",
"refId": "E"
},
{
"expr": "rate([[app]]_fuseclient_OpMetaExtentsList_count{app=\"$app\",cluster=\"$cluster\", instance=\"$instance\"}[1m])",
"format": "time_series",
"intervalFactor": 1,
"legendFormat": "OpMetaExtentsList_count_{{instance}}",
"refId": "F"
},
{
"expr": "[[app]]_fuseclient_OpMetaExtentsList{app=\"$app\",cluster=\"$cluster\", instance=\"$instance\"}",
"format": "time_series",
"intervalFactor": 1,
"legendFormat": "OpMetaExtentsList_{{instance}}",
"refId": "G"
}
],
"thresholds": [],
"timeFrom": null,
"timeShift": null,
"title": "OpMetaExtentsList",
"tooltip": {
"shared": true,
"sort": 0,
"value_type": "individual"
},
"type": "graph",
"xaxis": {
"buckets": null,
"mode": "time",
"name": null,
"show": true,
"values": []
},
"yaxes": [
{
"format": "locale",
"label": null,
"logBase": 1,
"max": null,
"min": null,
"show": true
},
{
"format": "ops",
"label": null,
"logBase": 1,
"max": null,
"min": null,
"show": true
}
],
"yaxis": {
"align": false,
"alignLevel": null
}
},
{
"aliasColors": {},
"bars": false,
"dashLength": 10,
"dashes": false,
"datasource": null,
"fill": 1,
"gridPos": {
"h": 7,
"w": 8,
"x": 0,
"y": 20
},
"id": 69,
"legend": {
"avg": false,
"current": false,
"max": false,
"min": false,
"show": true,
"total": false,
"values": false
},
"lines": true,
"linewidth": 1,
"links": [],
"nullPointMode": "null",
"percentage": false,
"pointradius": 5,
"points": false,
"renderer": "flot",
"seriesOverrides": [
{
"alias": "/.*_count_.*/",
"yaxis": 2
}
],
"spaceLength": 10,
"stack": false,
"steppedLine": false,
"targets": [
{
"expr": "[[app]]_fuseclient_OpMetaExtentsAdd{app=~\"$app\", cluster=\"$cluster\", instance=\"$instance\"}",
"format": "time_series",
"intervalFactor": 1,
"legendFormat": "OpMetaExtentsAdd_{{instance}}",
"refId": "H"
},
{
"expr": "rate([[app]]_fuseclient_OpMetaExtentsAdd_count{app=~\"$app\", cluster=\"$cluster\", instance=\"$instance\"}[1m])",
"format": "time_series",
"intervalFactor": 1,
"legendFormat": "OpMetaExtentsAdd_count_{{instance}}",
"refId": "A"
},
{
"expr": "[[app]]_fuseclient_OpMetaExtentsList{instance=~\"$instance\"}",
"format": "time_series",
"hide": true,
"intervalFactor": 1,
"legendFormat": "GetExtents",
"refId": "I"
}
],
"thresholds": [],
"timeFrom": null,
"timeShift": null,
"title": "OpMetaExtentsAdd",
"tooltip": {
"shared": true,
"sort": 0,
"value_type": "individual"
},
"type": "graph",
"xaxis": {
"buckets": null,
"mode": "time",
"name": null,
"show": true,
"values": []
},
"yaxes": [
{
"format": "ns",
"label": null,
"logBase": 1,
"max": null,
"min": null,
"show": true
},
{
"format": "ops",
"label": null,
"logBase": 1,
"max": null,
"min": null,
"show": true
}
],
"yaxis": {
"align": false,
"alignLevel": null
}
},
{
"aliasColors": {},
"bars": false,
"dashLength": 10,
"dashes": false,
"datasource": null,
"fill": 1,
"gridPos": {
"h": 7,
"w": 8,
"x": 8,
"y": 20
},
"id": 148,
"legend": {
"avg": false,
"current": false,
"max": false,
"min": false,
"show": true,
"total": false,
"values": false
},
"lines": true,
"linewidth": 1,
"links": [],
"nullPointMode": "null",
"percentage": false,
"pointradius": 5,
"points": false,
"renderer": "flot",
"seriesOverrides": [
{
"alias": "/.*_ops_.*/",
"yaxis": 2
}
],
"spaceLength": 10,
"stack": false,
"steppedLine": false,
"targets": [
{
"expr": "[[app]]_fuseclient_OpMetaOpen{app=\"$app\", cluster=\"$cluster\"}",
"format": "time_series",
"hide": true,
"intervalFactor": 1,
"legendFormat": "MetaOpen_{{instance}}",
"refId": "B"
},
{
"expr": "[[app]]_fuseclient_OpMetaReadDir{app=\"$app\", cluster=\"$cluster\"}",
"format": "time_series",
"hide": true,
"intervalFactor": 1,
"legendFormat": "MetaReaddir_{{instance}}",
"refId": "K"
},
{
"expr": "[[app]]_fuseclient_OpMetaSetattr{app=\"$app\", cluster=\"$cluster\"}",
"format": "time_series",
"hide": true,
"intervalFactor": 1,
"legendFormat": "MetaSetattr_{{instance}}",
"refId": "L"
},
{
"expr": "[[app]]_fuseclient_OpMetaLookup{app=\"$app\", cluster=\"$cluster\", instance=\"$instance\"}",
"format": "time_series",
"hide": false,
"intervalFactor": 1,
"legendFormat": "MetaLookup_{{instance}}",
"refId": "A"
},
{
"expr": "[[app]]_fuseclient_OpMetaTruncate{app=\"$app\", cluster=\"$cluster\"}",
"format": "time_series",
"hide": true,
"intervalFactor": 1,
"legendFormat": "MetaTruncate_{{instance}}",
"refId": "C"
},
{
"expr": "rate([[app]]_fuseclient_OpMetaOpen_count{app=\"$app\", cluster=\"$cluster\"}[1m])",
"format": "time_series",
"hide": true,
"intervalFactor": 1,
"legendFormat": "MetaOpen_ops_{{instance}}",
"refId": "D"
},
{
"expr": "rate([[app]]_fuseclient_OpMetaReadDir{app=\"$app\", cluster=\"$cluster\"}[1m])",
"format": "time_series",
"hide": true,
"intervalFactor": 1,
"legendFormat": "MetaReaddir_ops_{{instance}}",
"refId": "E"
},
{
"expr": "rate([[app]]_fuseclient_OpMetaSetattr{app=\"$app\", cluster=\"$cluster\"}[1m])",
"format": "time_series",
"hide": true,
"intervalFactor": 1,
"legendFormat": "MetaSetattr_ops_{{instance}}",
"refId": "F"
},
{
"expr": "rate([[app]]_fuseclient_OpMetaLookup_count{app=\"$app\", cluster=\"$cluster\", instance=\"$instance\"}[1m])",
"format": "time_series",
"hide": false,
"intervalFactor": 1,
"legendFormat": "MetaLookup_ops_{{instance}}",
"refId": "G"
},
{
"expr": "rate([[app]]_fuseclient_OpMetaTruncate{app=\"$app\", cluster=\"$cluster\"}[1m])",
"format": "time_series",
"hide": true,
"intervalFactor": 1,
"legendFormat": "MetaTruncate_ops_{{instance}}",
"refId": "H"
}
],
"thresholds": [],
"timeFrom": null,
"timeShift": null,
"title": "MetaLookup",
"tooltip": {
"shared": true,
"sort": 0,
"value_type": "individual"
},
"type": "graph",
"xaxis": {
"buckets": null,
"mode": "time",
"name": null,
"show": true,
"values": []
},
"yaxes": [
{
"format": "ns",
"label": null,
"logBase": 1,
"max": null,
"min": null,
"show": true
},
{
"format": "ops",
"label": null,
"logBase": 1,
"max": null,
"min": null,
"show": true
}
],
"yaxis": {
"align": false,
"alignLevel": null
}
},
{
"aliasColors": {},
"bars": false,
"dashLength": 10,
"dashes": false,
"datasource": null,
"fill": 1,
"gridPos": {
"h": 7,
"w": 8,
"x": 16,
"y": 20
},
"id": 67,
"legend": {
"avg": false,
"current": false,
"max": false,
"min": false,
"show": true,
"total": false,
"values": false
},
"lines": true,
"linewidth": 1,
"links": [],
"nullPointMode": "null",
"percentage": false,
"pointradius": 5,
"points": false,
"renderer": "flot",
"seriesOverrides": [
{
"alias": "/.*_count_.*/",
"yaxis": 2
}
],
"spaceLength": 10,
"stack": false,
"steppedLine": false,
"targets": [
{
"expr": "[[app]]_fuseclient_OpMetaBatchInodeGet{app=~\"$app\", cluster=\"$cluster\", instance=\"$instance\"}",
"format": "time_series",
"hide": false,
"intervalFactor": 1,
"legendFormat": "OpMetaBatchInodeGet_{{instance}}",
"refId": "A"
},
{
"expr": "[[app]]_fuseclient_OpMetaCreateInode{app=~\"$app\", cluster=\"$cluster\"}",
"format": "time_series",
"hide": true,
"intervalFactor": 1,
"legendFormat": "CreateInode_{{instance}}",
"refId": "D"
},
{
"expr": "[[app]]_fuseclient_OpMetaDeleteInode{app=~\"$app\", cluster=\"$cluster\"}",
"format": "time_series",
"hide": true,
"intervalFactor": 1,
"legendFormat": "DeleteInode_{{instance}}",
"refId": "F"
},
{
"expr": "[[app]]_fuseclient_OpMetaEvictInode{app=~\"$app\", cluster=\"$cluster\"}",
"format": "time_series",
"hide": true,
"intervalFactor": 1,
"legendFormat": "EvictInode_{{instance}}",
"refId": "G"
},
{
"expr": "[[app]]_fuseclient_OpMetaInodeGet{app=~\"$app\", cluster=\"$cluster\"}",
"format": "time_series",
"hide": true,
"intervalFactor": 1,
"legendFormat": "InodeGet_{{instance}}",
"refId": "J"
},
{
"expr": "[[app]]_fuseclient_OpMetaTruncate{app=~\"$app\", cluster=\"$cluster\"}",
"format": "time_series",
"hide": true,
"intervalFactor": 1,
"legendFormat": "Truncate_{{instance}}",
"refId": "B"
},
{
"expr": "rate([[app]]_fuseclient_OpMetaBatchInodeGet_count{app=~\"$app\", cluster=\"$cluster\", instance=\"$instance\"}[1m])",
"format": "time_series",
"hide": false,
"intervalFactor": 1,
"legendFormat": "OpMetaBatchInodeGet_count_{{instance}}",
"refId": "C"
},
{
"expr": "rate([[app]]_fuseclient_OpMetaCreateInode{app=~\"$app\", cluster=\"$cluster\"}[1m])",
"format": "time_series",
"hide": true,
"intervalFactor": 1,
"legendFormat": "CreateInode_count_{{instance}}",
"refId": "E"
},
{
"expr": "rate([[app]]_fuseclient_OpMetaDeleteInode{app=~\"$app\", cluster=\"$cluster\"}[1m])",
"format": "time_series",
"hide": true,
"intervalFactor": 1,
"legendFormat": "DeleteInode_count_{{instance}}",
"refId": "H"
},
{
"expr": "rate([[app]]_fuseclient_OpMetaEvictInode{app=~\"$app\", cluster=\"$cluster\"}[1m])",
"format": "time_series",
"hide": true,
"intervalFactor": 1,
"legendFormat": "EvictInode_count_{{instance}}",
"refId": "I"
},
{
"expr": "rate([[app]]_fuseclient_OpMetaInodeGet{app=~\"$app\", cluster=\"$cluster\"}[1m])",
"format": "time_series",
"hide": true,
"intervalFactor": 1,
"legendFormat": "InodeGet_count_{{instance}}",
"refId": "K"
},
{
"expr": "rate([[app]]_fuseclient_OpMetaTruncate{app=~\"$app\", cluster=\"$cluster\"}[1m])",
"format": "time_series",
"hide": true,
"intervalFactor": 1,
"legendFormat": "Truncate_count_{{instance}}",
"refId": "L"
}
],
"thresholds": [],
"timeFrom": null,
"timeShift": null,
"title": "OpMetaBatchInodeGet",
"tooltip": {
"shared": true,
"sort": 0,
"value_type": "individual"
},
"type": "graph",
"xaxis": {
"buckets": null,
"mode": "time",
"name": null,
"show": true,
"values": []
},
"yaxes": [
{
"format": "ns",
"label": null,
"logBase": 1,
"max": null,
"min": null,
"show": true
},
{
"format": "ops",
"label": null,
"logBase": 1,
"max": null,
"min": null,
"show": true
}
],
"yaxis": {
"align": false,
"alignLevel": null
}
},
{
"aliasColors": {},
"bars": false,
"dashLength": 10,
"dashes": false,
"datasource": "default",
"fill": 1,
"gridPos": {
"h": 7,
"w": 8,
"x": 0,
"y": 27
},
"id": 152,
"legend": {
"avg": false,
"current": false,
"max": false,
"min": false,
"show": true,
"total": false,
"values": false
},
"lines": true,
"linewidth": 1,
"links": [],
"nullPointMode": "null",
"percentage": false,
"pointradius": 5,
"points": false,
"renderer": "flot",
"seriesOverrides": [
{
"alias": "/.*_count_.*/",
"yaxis": 2
}
],
"spaceLength": 10,
"stack": false,
"steppedLine": false,
"targets": [
{
"expr": "[[app]]_fuseclient_filecreate{app=~\"$app\",cluster=\"$cluster\", instance=\"$instance\"}",
"format": "time_series",
"intervalFactor": 1,
"legendFormat": "filecreate_{{instance}}",
"refId": "A"
},
{
"expr": "rate([[app]]_fuseclient_filecreate_count{app=~\"$app\",cluster=\"$cluster\", instance=\"$instance\"}[1m])",
"format": "time_series",
"intervalFactor": 1,
"legendFormat": "filecreate_count_{{instance}}",
"refId": "B"
},
{
"expr": "[[app]]_fuseclient_filesync{app=~\"$app\",cluster=\"$cluster\"}",
"format": "time_series",
"hide": true,
"intervalFactor": 1,
"legendFormat": "filesync_{{instance}}",
"refId": "C"
},
{
"expr": "rate([[app]]_fuseclient_filesync_count{app=~\"$app\",cluster=\"$cluster\"}[1m])",
"format": "time_series",
"hide": true,
"intervalFactor": 1,
"legendFormat": "filesync_count_{{instance}}",
"refId": "D"
},
{
"expr": "[[app]]_fuseclient_filewrite{app=~\"$app\",cluster=\"$cluster\"}",
"format": "time_series",
"hide": true,
"intervalFactor": 1,
"legendFormat": "filewrite_{{instance}}",
"refId": "E"
},
{
"expr": "rate([[app]]_fuseclient_filewrite_count{app=~\"$app\",cluster=\"$cluster\"}[1m])",
"format": "time_series",
"hide": true,
"intervalFactor": 1,
"legendFormat": "filewrite_count_{{instance}}",
"refId": "F"
}
],
"thresholds": [],
"timeFrom": null,
"timeShift": null,
"title": "filecreate",
"tooltip": {
"shared": true,
"sort": 0,
"value_type": "individual"
},
"type": "graph",
"xaxis": {
"buckets": null,
"mode": "time",
"name": null,
"show": true,
"values": []
},
"yaxes": [
{
"format": "ns",
"label": null,
"logBase": 1,
"max": null,
"min": null,
"show": true
},
{
"format": "ops",
"label": null,
"logBase": 1,
"max": null,
"min": null,
"show": true
}
],
"yaxis": {
"align": false,
"alignLevel": null
}
},
{
"aliasColors": {},
"bars": false,
"dashLength": 10,
"dashes": false,
"datasource": null,
"fill": 1,
"gridPos": {
"h": 7,
"w": 8,
"x": 8,
"y": 27
},
"id": 158,
"legend": {
"avg": false,
"current": false,
"max": false,
"min": false,
"show": true,
"total": false,
"values": false
},
"lines": true,
"linewidth": 1,
"links": [],
"nullPointMode": "null",
"percentage": false,
"pointradius": 5,
"points": false,
"renderer": "flot",
"seriesOverrides": [
{
"alias": "/.*_count_.*/",
"yaxis": 2
}
],
"spaceLength": 10,
"stack": false,
"steppedLine": false,
"targets": [
{
"expr": "[[app]]_fuseclient_OpMetaInodeGet{app=~\"$app\", cluster=\"$cluster\", instance=\"$instance\"}",
"format": "time_series",
"hide": false,
"intervalFactor": 1,
"legendFormat": "InodeGet_{{instance}}",
"refId": "J"
},
{
"expr": "rate([[app]]_fuseclient_OpMetaInodeGet_count{app=~\"$app\", cluster=\"$cluster\", instance=\"$instance\"}[1m])",
"format": "time_series",
"hide": false,
"intervalFactor": 1,
"legendFormat": "InodeGet_count_{{instance}}",
"refId": "K"
}
],
"thresholds": [],
"timeFrom": null,
"timeShift": null,
"title": "InodeGet",
"tooltip": {
"shared": true,
"sort": 0,
"value_type": "individual"
},
"type": "graph",
"xaxis": {
"buckets": null,
"mode": "time",
"name": null,
"show": true,
"values": []
},
"yaxes": [
{
"format": "ns",
"label": null,
"logBase": 1,
"max": null,
"min": null,
"show": true
},
{
"format": "ops",
"label": null,
"logBase": 1,
"max": null,
"min": null,
"show": true
}
],
"yaxis": {
"align": false,
"alignLevel": null
}
},
{
"aliasColors": {},
"bars": false,
"dashLength": 10,
"dashes": false,
"datasource": null,
"fill": 1,
"gridPos": {
"h": 7,
"w": 8,
"x": 16,
"y": 27
},
"id": 155,
"legend": {
"avg": false,
"current": false,
"max": false,
"min": false,
"show": true,
"total": false,
"values": false
},
"lines": true,
"linewidth": 1,
"links": [],
"nullPointMode": "null",
"percentage": false,
"pointradius": 5,
"points": false,
"renderer": "flot",
"seriesOverrides": [
{
"alias": "/.*_count_.*/",
"yaxis": 2
}
],
"spaceLength": 10,
"stack": false,
"steppedLine": false,
"targets": [
{
"expr": "[[app]]_fuseclient_OpMetaBatchInodeGet{app=~\"$app\", cluster=\"$cluster\"}",
"format": "time_series",
"hide": true,
"intervalFactor": 1,
"legendFormat": "OpMetaBatchInodeGet_{{instance}}",
"refId": "A"
},
{
"expr": "[[app]]_fuseclient_OpMetaCreateInode{app=~\"$app\", cluster=\"$cluster\"}",
"format": "time_series",
"hide": true,
"intervalFactor": 1,
"legendFormat": "CreateInode_{{instance}}",
"refId": "D"
},
{
"expr": "[[app]]_fuseclient_OpMetaDeleteInode{app=~\"$app\", cluster=\"$cluster\"}",
"format": "time_series",
"hide": true,
"intervalFactor": 1,
"legendFormat": "DeleteInode_{{instance}}",
"refId": "F"
},
{
"expr": "[[app]]_fuseclient_OpMetaEvictInode{app=~\"$app\", cluster=\"$cluster\", instance=\"$instance\"}",
"format": "time_series",
"hide": false,
"intervalFactor": 1,
"legendFormat": "EvictInode_{{instance}}",
"refId": "G"
},
{
"expr": "[[app]]_fuseclient_OpMetaInodeGet{app=~\"$app\", cluster=\"$cluster\"}",
"format": "time_series",
"hide": true,
"intervalFactor": 1,
"legendFormat": "InodeGet_{{instance}}",
"refId": "J"
},
{
"expr": "[[app]]_fuseclient_OpMetaTruncate{app=~\"$app\", cluster=\"$cluster\"}",
"format": "time_series",
"hide": true,
"intervalFactor": 1,
"legendFormat": "Truncate_{{instance}}",
"refId": "B"
},
{
"expr": "rate([[app]]_fuseclient_OpMetaBatchInodeGet_count{app=~\"$app\", cluster=\"$cluster\"}[1m])",
"format": "time_series",
"hide": true,
"intervalFactor": 1,
"legendFormat": "OpMetaBatchInodeGet_count_{{instance}}",
"refId": "C"
},
{
"expr": "rate([[app]]_fuseclient_OpMetaCreateInode{app=~\"$app\", cluster=\"$cluster\"}[1m])",
"format": "time_series",
"hide": true,
"intervalFactor": 1,
"legendFormat": "CreateInode_count_{{instance}}",
"refId": "E"
},
{
"expr": "rate([[app]]_fuseclient_OpMetaDeleteInode{app=~\"$app\", cluster=\"$cluster\"}[1m])",
"format": "time_series",
"hide": true,
"intervalFactor": 1,
"legendFormat": "DeleteInode_count_{{instance}}",
"refId": "H"
},
{
"expr": "rate([[app]]_fuseclient_OpMetaEvictInode_count{app=~\"$app\", cluster=\"$cluster\", instance=\"$instance\"}[1m])",
"format": "time_series",
"hide": false,
"intervalFactor": 1,
"legendFormat": "EvictInode_count_{{instance}}",
"refId": "I"
},
{
"expr": "rate([[app]]_fuseclient_OpMetaInodeGet{app=~\"$app\", cluster=\"$cluster\"}[1m])",
"format": "time_series",
"hide": true,
"intervalFactor": 1,
"legendFormat": "InodeGet_count_{{instance}}",
"refId": "K"
},
{
"expr": "rate([[app]]_fuseclient_OpMetaTruncate{app=~\"$app\", cluster=\"$cluster\"}[1m])",
"format": "time_series",
"hide": true,
"intervalFactor": 1,
"legendFormat": "Truncate_count_{{instance}}",
"refId": "L"
}
],
"thresholds": [],
"timeFrom": null,
"timeShift": null,
"title": "EvictInode",
"tooltip": {
"shared": true,
"sort": 0,
"value_type": "individual"
},
"type": "graph",
"xaxis": {
"buckets": null,
"mode": "time",
"name": null,
"show": true,
"values": []
},
"yaxes": [
{
"format": "ns",
"label": null,
"logBase": 1,
"max": null,
"min": null,
"show": true
},
{
"format": "ops",
"label": null,
"logBase": 1,
"max": null,
"min": null,
"show": true
}
],
"yaxis": {
"align": false,
"alignLevel": null
}
},
{
"aliasColors": {},
"bars": false,
"dashLength": 10,
"dashes": false,
"datasource": "default",
"fill": 1,
"gridPos": {
"h": 7,
"w": 8,
"x": 0,
"y": 34
},
"id": 153,
"legend": {
"avg": false,
"current": false,
"max": false,
"min": false,
"show": true,
"total": false,
"values": false
},
"lines": true,
"linewidth": 1,
"links": [],
"nullPointMode": "null",
"percentage": false,
"pointradius": 5,
"points": false,
"renderer": "flot",
"seriesOverrides": [
{
"alias": "/.*_count_.*/",
"yaxis": 2
}
],
"spaceLength": 10,
"stack": false,
"steppedLine": false,
"targets": [
{
"expr": "[[app]]_fuseclient_filecreate{app=~\"$app\",cluster=\"$cluster\"}",
"format": "time_series",
"hide": true,
"intervalFactor": 1,
"legendFormat": "filecreate_{{instance}}",
"refId": "A"
},
{
"expr": "rate([[app]]_fuseclient_filecreate_count{app=~\"$app\",cluster=\"$cluster\"}[1m])",
"format": "time_series",
"hide": true,
"intervalFactor": 1,
"legendFormat": "filecreate_count_{{instance}}",
"refId": "B"
},
{
"expr": "[[app]]_fuseclient_filesync{app=~\"$app\",cluster=\"$cluster\", instance=\"$instance\"}",
"format": "time_series",
"hide": false,
"intervalFactor": 1,
"legendFormat": "filesync_{{instance}}",
"refId": "C"
},
{
"expr": "rate([[app]]_fuseclient_filesync_count{app=~\"$app\",cluster=\"$cluster\", instance=\"$instance\"}[1m])",
"format": "time_series",
"hide": false,
"intervalFactor": 1,
"legendFormat": "filesync_count_{{instance}}",
"refId": "D"
},
{
"expr": "[[app]]_fuseclient_filewrite{app=~\"$app\",cluster=\"$cluster\"}",
"format": "time_series",
"hide": true,
"intervalFactor": 1,
"legendFormat": "filewrite_{{instance}}",
"refId": "E"
},
{
"expr": "rate([[app]]_fuseclient_filewrite_count{app=~\"$app\",cluster=\"$cluster\"}[1m])",
"format": "time_series",
"hide": true,
"intervalFactor": 1,
"legendFormat": "filewrite_count_{{instance}}",
"refId": "F"
}
],
"thresholds": [],
"timeFrom": null,
"timeShift": null,
"title": "filesync",
"tooltip": {
"shared": true,
"sort": 0,
"value_type": "individual"
},
"type": "graph",
"xaxis": {
"buckets": null,
"mode": "time",
"name": null,
"show": true,
"values": []
},
"yaxes": [
{
"format": "ns",
"label": null,
"logBase": 1,
"max": null,
"min": null,
"show": true
},
{
"format": "ops",
"label": null,
"logBase": 1,
"max": null,
"min": null,
"show": true
}
],
"yaxis": {
"align": false,
"alignLevel": null
}
},
{
"aliasColors": {},
"bars": false,
"dashLength": 10,
"dashes": false,
"datasource": "default",
"fill": 1,
"gridPos": {
"h": 7,
"w": 8,
"x": 8,
"y": 34
},
"id": 100,
"legend": {
"avg": false,
"current": false,
"max": false,
"min": false,
"show": true,
"total": false,
"values": false
},
"lines": true,
"linewidth": 1,
"links": [],
"nullPointMode": "null",
"percentage": false,
"pointradius": 5,
"points": false,
"renderer": "flot",
"seriesOverrides": [
{
"alias": "/.*_count_.*/",
"yaxis": 2
}
],
"spaceLength": 10,
"stack": false,
"steppedLine": false,
"targets": [
{
"expr": "[[app]]_fuseclient_fileread{app=~\"$app\",cluster=\"$cluster\", instance=\"$instance\"}",
"format": "time_series",
"intervalFactor": 1,
"legendFormat": "fileread_{{instance}}",
"refId": "A"
},
{
"expr": "rate([[app]]_fuseclient_fileread_count{app=~\"$app\",cluster=\"$cluster\", instance=\"$instance\"}[1m])",
"format": "time_series",
"intervalFactor": 1,
"legendFormat": "fileread_count_{{instance}}",
"refId": "B"
}
],
"thresholds": [],
"timeFrom": null,
"timeShift": null,
"title": "fileread",
"tooltip": {
"shared": true,
"sort": 0,
"value_type": "individual"
},
"type": "graph",
"xaxis": {
"buckets": null,
"mode": "time",
"name": null,
"show": true,
"values": []
},
"yaxes": [
{
"format": "ns",
"label": null,
"logBase": 1,
"max": null,
"min": null,
"show": true
},
{
"format": "ops",
"label": null,
"logBase": 1,
"max": null,
"min": null,
"show": true
}
],
"yaxis": {
"align": false,
"alignLevel": null
}
},
{
"aliasColors": {},
"bars": false,
"dashLength": 10,
"dashes": false,
"datasource": null,
"fill": 1,
"gridPos": {
"h": 7,
"w": 8,
"x": 16,
"y": 34
},
"id": 150,
"legend": {
"avg": false,
"current": false,
"max": false,
"min": false,
"show": true,
"total": false,
"values": false
},
"lines": true,
"linewidth": 1,
"links": [],
"nullPointMode": "null",
"percentage": false,
"pointradius": 5,
"points": false,
"renderer": "flot",
"seriesOverrides": [
{
"alias": "/.*_ops_.*/",
"yaxis": 2
}
],
"spaceLength": 10,
"stack": false,
"steppedLine": false,
"targets": [
{
"expr": "[[app]]_fuseclient_OpMetaOpen{app=\"$app\", cluster=\"$cluster\", instance=\"$instance\"}",
"format": "time_series",
"hide": false,
"intervalFactor": 1,
"legendFormat": "MetaOpen_{{instance}}",
"refId": "B"
},
{
"expr": "[[app]]_fuseclient_OpMetaReadDir{app=\"$app\", cluster=\"$cluster\"}",
"format": "time_series",
"hide": true,
"intervalFactor": 1,
"legendFormat": "MetaReaddir_{{instance}}",
"refId": "K"
},
{
"expr": "[[app]]_fuseclient_OpMetaSetattr{app=\"$app\", cluster=\"$cluster\"}",
"format": "time_series",
"hide": true,
"intervalFactor": 1,
"legendFormat": "MetaSetattr_{{instance}}",
"refId": "L"
},
{
"expr": "[[app]]_fuseclient_OpMetaLookup{app=\"$app\", cluster=\"$cluster\"}",
"format": "time_series",
"hide": true,
"intervalFactor": 1,
"legendFormat": "MetaLookup_{{instance}}",
"refId": "A"
},
{
"expr": "[[app]]_fuseclient_OpMetaTruncate{app=\"$app\", cluster=\"$cluster\"}",
"format": "time_series",
"hide": true,
"intervalFactor": 1,
"legendFormat": "MetaTruncate_{{instance}}",
"refId": "C"
},
{
"expr": "rate([[app]]_fuseclient_OpMetaOpen_count{app=\"$app\", cluster=\"$cluster\", instance=\"$instance\"}[1m])",
"format": "time_series",
"hide": false,
"intervalFactor": 1,
"legendFormat": "MetaOpen_ops_{{instance}}",
"refId": "D"
},
{
"expr": "rate([[app]]_fuseclient_OpMetaReadDir{app=\"$app\", cluster=\"$cluster\"}[1m])",
"format": "time_series",
"hide": true,
"intervalFactor": 1,
"legendFormat": "MetaReaddir_ops_{{instance}}",
"refId": "E"
},
{
"expr": "rate([[app]]_fuseclient_OpMetaSetattr{app=\"$app\", cluster=\"$cluster\"}[1m])",
"format": "time_series",
"hide": true,
"intervalFactor": 1,
"legendFormat": "MetaSetattr_ops_{{instance}}",
"refId": "F"
},
{
"expr": "rate([[app]]_fuseclient_OpMetaLookup{app=\"$app\", cluster=\"$cluster\"}[1m])",
"format": "time_series",
"hide": true,
"intervalFactor": 1,
"legendFormat": "MetaLookup_ops_{{instance}}",
"refId": "G"
},
{
"expr": "rate([[app]]_fuseclient_OpMetaTruncate{app=\"$app\", cluster=\"$cluster\"}[1m])",
"format": "time_series",
"hide": true,
"intervalFactor": 1,
"legendFormat": "MetaTruncate_ops_{{instance}}",
"refId": "H"
}
],
"thresholds": [],
"timeFrom": null,
"timeShift": null,
"title": "MetaOpen",
"tooltip": {
"shared": true,
"sort": 0,
"value_type": "individual"
},
"type": "graph",
"xaxis": {
"buckets": null,
"mode": "time",
"name": null,
"show": true,
"values": []
},
"yaxes": [
{
"format": "ns",
"label": null,
"logBase": 1,
"max": null,
"min": null,
"show": true
},
{
"format": "ops",
"label": null,
"logBase": 1,
"max": null,
"min": null,
"show": true
}
],
"yaxis": {
"align": false,
"alignLevel": null
}
},
{
"aliasColors": {},
"bars": false,
"dashLength": 10,
"dashes": false,
"datasource": null,
"fill": 1,
"gridPos": {
"h": 7,
"w": 8,
"x": 0,
"y": 41
},
"id": 147,
"legend": {
"avg": false,
"current": false,
"max": false,
"min": false,
"show": true,
"total": false,
"values": false
},
"lines": true,
"linewidth": 1,
"links": [],
"nullPointMode": "null",
"percentage": false,
"pointradius": 5,
"points": false,
"renderer": "flot",
"seriesOverrides": [
{
"alias": "/.*_ops_.*/",
"yaxis": 2
}
],
"spaceLength": 10,
"stack": false,
"steppedLine": false,
"targets": [
{
"expr": "[[app]]_fuseclient_OpMetaOpen{app=\"$app\", cluster=\"$cluster\"}",
"format": "time_series",
"hide": true,
"intervalFactor": 1,
"legendFormat": "MetaOpen_{{instance}}",
"refId": "B"
},
{
"expr": "[[app]]_fuseclient_OpMetaReadDir{app=\"$app\", cluster=\"$cluster\"}",
"format": "time_series",
"hide": true,
"intervalFactor": 1,
"legendFormat": "MetaReaddir_{{instance}}",
"refId": "K"
},
{
"expr": "[[app]]_fuseclient_OpMetaSetattr{app=\"$app\", cluster=\"$cluster\"}",
"format": "time_series",
"hide": true,
"intervalFactor": 1,
"legendFormat": "MetaSetattr_{{instance}}",
"refId": "L"
},
{
"expr": "[[app]]_fuseclient_OpMetaLookup{app=\"$app\", cluster=\"$cluster\"}",
"format": "time_series",
"hide": true,
"intervalFactor": 1,
"legendFormat": "MetaLookup_{{instance}}",
"refId": "A"
},
{
"expr": "[[app]]_fuseclient_OpMetaTruncate{app=\"$app\", cluster=\"$cluster\", instance=\"$instance\"}",
"format": "time_series",
"hide": false,
"intervalFactor": 1,
"legendFormat": "MetaTruncate_{{instance}}",
"refId": "C"
},
{
"expr": "rate([[app]]_fuseclient_OpMetaOpen_count{app=\"$app\", cluster=\"$cluster\"}[1m])",
"format": "time_series",
"hide": true,
"intervalFactor": 1,
"legendFormat": "MetaOpen_ops_{{instance}}",
"refId": "D"
},
{
"expr": "rate([[app]]_fuseclient_OpMetaReadDir{app=\"$app\", cluster=\"$cluster\"}[1m])",
"format": "time_series",
"hide": true,
"intervalFactor": 1,
"legendFormat": "MetaReaddir_ops_{{instance}}",
"refId": "E"
},
{
"expr": "rate([[app]]_fuseclient_OpMetaSetattr{app=\"$app\", cluster=\"$cluster\"}[1m])",
"format": "time_series",
"hide": true,
"intervalFactor": 1,
"legendFormat": "MetaSetattr_ops_{{instance}}",
"refId": "F"
},
{
"expr": "rate([[app]]_fuseclient_OpMetaLookup{app=\"$app\", cluster=\"$cluster\"}[1m])",
"format": "time_series",
"hide": true,
"intervalFactor": 1,
"legendFormat": "MetaLookup_ops_{{instance}}",
"refId": "G"
},
{
"expr": "rate([[app]]_fuseclient_OpMetaTruncate_count{app=\"$app\", cluster=\"$cluster\", instance=\"$instance\"}[1m])",
"format": "time_series",
"hide": false,
"intervalFactor": 1,
"legendFormat": "MetaTruncate_ops_{{instance}}",
"refId": "H"
}
],
"thresholds": [],
"timeFrom": null,
"timeShift": null,
"title": "MetaTruncate",
"tooltip": {
"shared": true,
"sort": 0,
"value_type": "individual"
},
"type": "graph",
"xaxis": {
"buckets": null,
"mode": "time",
"name": null,
"show": true,
"values": []
},
"yaxes": [
{
"format": "ns",
"label": null,
"logBase": 1,
"max": null,
"min": null,
"show": true
},
{
"format": "ops",
"label": null,
"logBase": 1,
"max": null,
"min": null,
"show": true
}
],
"yaxis": {
"align": false,
"alignLevel": null
}
},
{
"aliasColors": {},
"bars": false,
"dashLength": 10,
"dashes": false,
"datasource": "default",
"fill": 1,
"gridPos": {
"h": 7,
"w": 8,
"x": 8,
"y": 41
},
"id": 154,
"legend": {
"avg": false,
"current": false,
"max": false,
"min": false,
"show": true,
"total": false,
"values": false
},
"lines": true,
"linewidth": 1,
"links": [],
"nullPointMode": "null",
"percentage": false,
"pointradius": 5,
"points": false,
"renderer": "flot",
"seriesOverrides": [
{
"alias": "/.*_count_.*/",
"yaxis": 2
}
],
"spaceLength": 10,
"stack": false,
"steppedLine": false,
"targets": [
{
"expr": "[[app]]_fuseclient_filecreate{app=~\"$app\",cluster=\"$cluster\"}",
"format": "time_series",
"hide": true,
"intervalFactor": 1,
"legendFormat": "filecreate_{{instance}}",
"refId": "A"
},
{
"expr": "rate([[app]]_fuseclient_filecreate_count{app=~\"$app\",cluster=\"$cluster\"}[1m])",
"format": "time_series",
"hide": true,
"intervalFactor": 1,
"legendFormat": "filecreate_count_{{instance}}",
"refId": "B"
},
{
"expr": "[[app]]_fuseclient_filesync{app=~\"$app\",cluster=\"$cluster\"}",
"format": "time_series",
"hide": true,
"intervalFactor": 1,
"legendFormat": "filesync_{{instance}}",
"refId": "C"
},
{
"expr": "rate([[app]]_fuseclient_filesync_count{app=~\"$app\",cluster=\"$cluster\"}[1m])",
"format": "time_series",
"hide": true,
"intervalFactor": 1,
"legendFormat": "filesync_count_{{instance}}",
"refId": "D"
},
{
"expr": "[[app]]_fuseclient_filewrite{app=~\"$app\",cluster=\"$cluster\", instance=\"$instance\"}",
"format": "time_series",
"hide": false,
"intervalFactor": 1,
"legendFormat": "filewrite_{{instance}}",
"refId": "E"
},
{
"expr": "rate([[app]]_fuseclient_filewrite_count{app=~\"$app\",cluster=\"$cluster\", instance=\"$instance\"}[1m])",
"format": "time_series",
"hide": false,
"intervalFactor": 1,
"legendFormat": "filewrite_count_{{instance}}",
"refId": "F"
}
],
"thresholds": [],
"timeFrom": null,
"timeShift": null,
"title": "filewrite",
"tooltip": {
"shared": true,
"sort": 0,
"value_type": "individual"
},
"type": "graph",
"xaxis": {
"buckets": null,
"mode": "time",
"name": null,
"show": true,
"values": []
},
"yaxes": [
{
"format": "ns",
"label": null,
"logBase": 1,
"max": null,
"min": null,
"show": true
},
{
"format": "ops",
"label": null,
"logBase": 1,
"max": null,
"min": null,
"show": true
}
],
"yaxis": {
"align": false,
"alignLevel": null
}
},
{
"aliasColors": {},
"bars": false,
"dashLength": 10,
"dashes": false,
"datasource": null,
"fill": 1,
"gridPos": {
"h": 7,
"w": 8,
"x": 16,
"y": 41
},
"id": 98,
"legend": {
"avg": false,
"current": false,
"max": false,
"min": false,
"show": true,
"total": false,
"values": false
},
"lines": true,
"linewidth": 1,
"links": [],
"nullPointMode": "null",
"percentage": false,
"pointradius": 5,
"points": false,
"renderer": "flot",
"seriesOverrides": [],
"spaceLength": 10,
"stack": false,
"steppedLine": false,
"targets": [
{
"expr": "[[app]]_fuseclient_StreamWrite{instance=~\"$instance\",cluster=\"$cluster\", instance=\"$instance\"}",
"format": "time_series",
"intervalFactor": 1,
"legendFormat": "Write",
"refId": "A"
},
{
"expr": "[[app]]_fuseclient_StreamRead{instance=~\"$instance\",cluster=\"$cluster\", instance=\"$instance\"}",
"format": "time_series",
"intervalFactor": 1,
"legendFormat": "Read",
"refId": "C"
},
{
"expr": "[[app]]_fuseclient_StreamOverwrite{instance=~\"$instance\",cluster=\"$cluster\", instance=\"$instance\"}",
"format": "time_series",
"intervalFactor": 1,
"legendFormat": "Overwrite",
"refId": "D"
},
{
"expr": "[[app]]_fuseclient_StreamCreateExtent{instance=~\"$instance\",cluster=\"$cluster\", instance=\"$instance\"}",
"format": "time_series",
"intervalFactor": 1,
"legendFormat": "CreateExtent",
"refId": "B"
},
{
"expr": "[[app]]_fuseclient_StreamSyncWrite{instance=~\"$instance\",cluster=\"$cluster\"}",
"format": "time_series",
"intervalFactor": 1,
"legendFormat": "SyncWrite",
"refId": "E"
},
{
"expr": "[[app]]_fuseclient_StreamSyncOverwrite{instance=~\"$instance\",cluster=\"$cluster\"}",
"format": "time_series",
"intervalFactor": 1,
"legendFormat": "SyncOverwrite",
"refId": "F"
}
],
"thresholds": [],
"timeFrom": null,
"timeShift": null,
"title": "fuseclient_StreamSDK_latency",
"tooltip": {
"shared": true,
"sort": 0,
"value_type": "individual"
},
"type": "graph",
"xaxis": {
"buckets": null,
"mode": "time",
"name": null,
"show": true,
"values": []
},
"yaxes": [
{
"format": "ns",
"label": null,
"logBase": 1,
"max": null,
"min": null,
"show": true
},
{
"format": "short",
"label": null,
"logBase": 1,
"max": null,
"min": null,
"show": true
}
],
"yaxis": {
"align": false,
"alignLevel": null
}
},
{
"aliasColors": {},
"bars": false,
"dashLength": 10,
"dashes": false,
"datasource": null,
"fill": 1,
"gridPos": {
"h": 7,
"w": 8,
"x": 0,
"y": 48
},
"id": 106,
"legend": {
"avg": false,
"current": false,
"max": false,
"min": false,
"show": true,
"total": false,
"values": false
},
"lines": true,
"linewidth": 1,
"links": [],
"nullPointMode": "null",
"percentage": false,
"pointradius": 5,
"points": false,
"renderer": "flot",
"seriesOverrides": [],
"spaceLength": 10,
"stack": false,
"steppedLine": false,
"targets": [
{
"expr": "[[app]]_fuseclient_FlushInRead{app=\"$app\", cluster=\"$cluster\", instance=\"$instance\"}",
"format": "time_series",
"intervalFactor": 1,
"legendFormat": "FlushInRead_{{instance}}",
"refId": "A"
},
{
"expr": "[[app]]_fuseclient_StreamPrepareReadRequests{app=~\"$app\", cluster=\"$cluster\", instance=\"$instance\"}",
"format": "time_series",
"hide": true,
"intervalFactor": 1,
"legendFormat": "PrepareReadRequest_{{instance}}",
"refId": "B"
}
],
"thresholds": [],
"timeFrom": null,
"timeShift": null,
"title": "FlushInRead",
"tooltip": {
"shared": true,
"sort": 0,
"value_type": "individual"
},
"type": "graph",
"xaxis": {
"buckets": null,
"mode": "time",
"name": null,
"show": true,
"values": []
},
"yaxes": [
{
"format": "ns",
"label": null,
"logBase": 1,
"max": null,
"min": null,
"show": true
},
{
"format": "short",
"label": null,
"logBase": 1,
"max": null,
"min": null,
"show": true
}
],
"yaxis": {
"align": false,
"alignLevel": null
}
},
{
"aliasColors": {},
"bars": false,
"dashLength": 10,
"dashes": false,
"datasource": null,
"fill": 1,
"gridPos": {
"h": 7,
"w": 8,
"x": 8,
"y": 48
},
"id": 149,
"legend": {
"avg": false,
"current": false,
"max": false,
"min": false,
"show": true,
"total": false,
"values": false
},
"lines": true,
"linewidth": 1,
"links": [],
"nullPointMode": "null",
"percentage": false,
"pointradius": 5,
"points": false,
"renderer": "flot",
"seriesOverrides": [
{
"alias": "/.*_ops_.*/",
"yaxis": 2
}
],
"spaceLength": 10,
"stack": false,
"steppedLine": false,
"targets": [
{
"expr": "[[app]]_fuseclient_OpMetaOpen{app=\"$app\", cluster=\"$cluster\"}",
"format": "time_series",
"hide": true,
"intervalFactor": 1,
"legendFormat": "MetaOpen_{{instance}}",
"refId": "B"
},
{
"expr": "[[app]]_fuseclient_OpMetaReadDir{app=\"$app\", cluster=\"$cluster\"}",
"format": "time_series",
"hide": true,
"intervalFactor": 1,
"legendFormat": "MetaReaddir_{{instance}}",
"refId": "K"
},
{
"expr": "[[app]]_fuseclient_OpMetaSetattr{app=\"$app\", cluster=\"$cluster\", instance=\"$instance\"}",
"format": "time_series",
"hide": false,
"intervalFactor": 1,
"legendFormat": "MetaSetattr_{{instance}}",
"refId": "L"
},
{
"expr": "[[app]]_fuseclient_OpMetaLookup{app=\"$app\", cluster=\"$cluster\"}",
"format": "time_series",
"hide": true,
"intervalFactor": 1,
"legendFormat": "MetaLookup_{{instance}}",
"refId": "A"
},
{
"expr": "[[app]]_fuseclient_OpMetaTruncate{app=\"$app\", cluster=\"$cluster\"}",
"format": "time_series",
"hide": true,
"intervalFactor": 1,
"legendFormat": "MetaTruncate_{{instance}}",
"refId": "C"
},
{
"expr": "rate([[app]]_fuseclient_OpMetaOpen_count{app=\"$app\", cluster=\"$cluster\"}[1m])",
"format": "time_series",
"hide": true,
"intervalFactor": 1,
"legendFormat": "MetaOpen_ops_{{instance}}",
"refId": "D"
},
{
"expr": "rate([[app]]_fuseclient_OpMetaReadDir{app=\"$app\", cluster=\"$cluster\"}[1m])",
"format": "time_series",
"hide": true,
"intervalFactor": 1,
"legendFormat": "MetaReaddir_ops_{{instance}}",
"refId": "E"
},
{
"expr": "rate([[app]]_fuseclient_OpMetaSetattr_cout{app=\"$app\", cluster=\"$cluster\", instance=\"$instance\"}[1m])",
"format": "time_series",
"hide": false,
"intervalFactor": 1,
"legendFormat": "MetaSetattr_ops_{{instance}}",
"refId": "F"
},
{
"expr": "rate([[app]]_fuseclient_OpMetaLookup_count{app=\"$app\", cluster=\"$cluster\"}[1m])",
"format": "time_series",
"hide": true,
"intervalFactor": 1,
"legendFormat": "MetaLookup_ops_{{instance}}",
"refId": "G"
},
{
"expr": "rate([[app]]_fuseclient_OpMetaTruncate_count{app=\"$app\", cluster=\"$cluster\"}[1m])",
"format": "time_series",
"hide": true,
"intervalFactor": 1,
"legendFormat": "MetaTruncate_ops_{{instance}}",
"refId": "H"
}
],
"thresholds": [],
"timeFrom": null,
"timeShift": null,
"title": "MetaSetattr",
"tooltip": {
"shared": true,
"sort": 0,
"value_type": "individual"
},
"type": "graph",
"xaxis": {
"buckets": null,
"mode": "time",
"name": null,
"show": true,
"values": []
},
"yaxes": [
{
"format": "ns",
"label": null,
"logBase": 1,
"max": null,
"min": null,
"show": true
},
{
"format": "ops",
"label": null,
"logBase": 1,
"max": null,
"min": null,
"show": true
}
],
"yaxis": {
"align": false,
"alignLevel": null
}
},
{
"aliasColors": {},
"bars": false,
"dashLength": 10,
"dashes": false,
"datasource": null,
"fill": 1,
"gridPos": {
"h": 7,
"w": 8,
"x": 16,
"y": 48
},
"id": 151,
"legend": {
"avg": false,
"current": false,
"max": false,
"min": false,
"show": true,
"total": false,
"values": false
},
"lines": true,
"linewidth": 1,
"links": [],
"nullPointMode": "null",
"percentage": false,
"pointradius": 5,
"points": false,
"renderer": "flot",
"seriesOverrides": [],
"spaceLength": 10,
"stack": false,
"steppedLine": false,
"targets": [
{
"expr": "[[app]]_fuseclient_FlushInRead{app=\"$app\", cluster=\"$cluster\", instance=\"$instance\"}",
"format": "time_series",
"hide": true,
"intervalFactor": 1,
"legendFormat": "FlushInRead_{{instance}}",
"refId": "A"
},
{
"expr": "[[app]]_fuseclient_PrepareReadRequest{app=~\"$app\", cluster=\"$cluster\", instance=\"$instance\"}",
"format": "time_series",
"hide": false,
"intervalFactor": 1,
"legendFormat": "PrepareReadRequest_{{instance}}",
"refId": "B"
}
],
"thresholds": [],
"timeFrom": null,
"timeShift": null,
"title": "PrepareReadRequest",
"tooltip": {
"shared": true,
"sort": 0,
"value_type": "individual"
},
"type": "graph",
"xaxis": {
"buckets": null,
"mode": "time",
"name": null,
"show": true,
"values": []
},
"yaxes": [
{
"format": "ns",
"label": null,
"logBase": 1,
"max": null,
"min": null,
"show": true
},
{
"format": "short",
"label": null,
"logBase": 1,
"max": null,
"min": null,
"show": true
}
],
"yaxis": {
"align": false,
"alignLevel": null
}
},
{
"aliasColors": {},
"bars": false,
"dashLength": 10,
"dashes": false,
"datasource": null,
"fill": 1,
"gridPos": {
"h": 7,
"w": 8,
"x": 0,
"y": 55
},
"id": 159,
"legend": {
"avg": false,
"current": false,
"max": false,
"min": false,
"show": true,
"total": false,
"values": false
},
"lines": true,
"linewidth": 1,
"links": [],
"nullPointMode": "null",
"percentage": false,
"pointradius": 5,
"points": false,
"renderer": "flot",
"seriesOverrides": [
{
"alias": "/.*_count_.*/",
"yaxis": 2
}
],
"spaceLength": 10,
"stack": false,
"steppedLine": false,
"targets": [
{
"expr": "[[app]]_fuseclient_OpMetaTruncate{app=~\"$app\", cluster=\"$cluster\", instance=\"$instance\"}",
"format": "time_series",
"hide": false,
"intervalFactor": 1,
"legendFormat": "Truncate_{{instance}}",
"refId": "B"
},
{
"expr": "rate([[app]]_fuseclient_OpMetaTruncate_count{app=~\"$app\", cluster=\"$cluster\", instance=\"$instance\"}[1m])",
"format": "time_series",
"hide": false,
"intervalFactor": 1,
"legendFormat": "Truncate_count_{{instance}}",
"refId": "L"
}
],
"thresholds": [],
"timeFrom": null,
"timeShift": null,
"title": "Truncate",
"tooltip": {
"shared": true,
"sort": 0,
"value_type": "individual"
},
"type": "graph",
"xaxis": {
"buckets": null,
"mode": "time",
"name": null,
"show": true,
"values": []
},
"yaxes": [
{
"format": "ns",
"label": null,
"logBase": 1,
"max": null,
"min": null,
"show": true
},
{
"format": "ops",
"label": null,
"logBase": 1,
"max": null,
"min": null,
"show": true
}
],
"yaxis": {
"align": false,
"alignLevel": null
}
},
{
"aliasColors": {},
"bars": false,
"dashLength": 10,
"dashes": false,
"datasource": null,
"fill": 1,
"gridPos": {
"h": 7,
"w": 8,
"x": 8,
"y": 55
},
"id": 161,
"legend": {
"avg": false,
"current": false,
"max": false,
"min": false,
"show": true,
"total": false,
"values": false
},
"lines": true,
"linewidth": 1,
"links": [],
"nullPointMode": "null",
"percentage": false,
"pointradius": 5,
"points": false,
"renderer": "flot",
"seriesOverrides": [
{
"alias": "/.*_count_.*/",
"yaxis": 2
}
],
"spaceLength": 10,
"stack": false,
"steppedLine": false,
"targets": [
{
"expr": "[[app]]_fuseclient_OpMetaCreateDentry{app=~\"$app\", cluster=\"$cluster\"}",
"format": "time_series",
"hide": true,
"intervalFactor": 1,
"legendFormat": "OpMetaCreateDentry_{{instance}}",
"refId": "C"
},
{
"expr": "[[app]]_fuseclient_OpMetaDeleteDentry{app=~\"$app\", cluster=\"$cluster\"}",
"format": "time_series",
"hide": true,
"intervalFactor": 1,
"legendFormat": "OpMetaDeleteDentry_{{instance}}",
"refId": "E"
},
{
"expr": "[[app]]_fuseclient_OpMetaUpdateDentry{app=~\"$app\", cluster=\"$cluster\", instance=\"$instance\"}",
"format": "time_series",
"hide": false,
"intervalFactor": 1,
"legendFormat": "OpMetaUpdateDentry_{{instance}}",
"refId": "A"
},
{
"expr": "[[app]]_fuseclient_OpMetaCreateDentry_count{app=~\"$app\", cluster=\"$cluster\"}",
"format": "time_series",
"hide": true,
"intervalFactor": 1,
"legendFormat": "OpMetaCreateDentry_count_{{instance}}",
"refId": "B"
},
{
"expr": "[[app]]_fuseclient_OpMetaDeleteDentry_count{app=~\"$app\", cluster=\"$cluster\"}",
"format": "time_series",
"hide": true,
"intervalFactor": 1,
"legendFormat": "OpMetaDeleteDentry_count_{{instance}}",
"refId": "D"
},
{
"expr": "[[app]]_fuseclient_OpMetaUpdateDentry_count{app=~\"$app\", cluster=\"$cluster\", instance=\"$instance\"}",
"format": "time_series",
"hide": false,
"intervalFactor": 1,
"legendFormat": "OpMetaUpdateDentry_count_{{instance}}",
"refId": "F"
}
],
"thresholds": [],
"timeFrom": null,
"timeShift": null,
"title": "OpMetaUpdateDentry",
"tooltip": {
"shared": true,
"sort": 0,
"value_type": "individual"
},
"type": "graph",
"xaxis": {
"buckets": null,
"mode": "time",
"name": null,
"show": true,
"values": []
},
"yaxes": [
{
"format": "ns",
"label": null,
"logBase": 1,
"max": null,
"min": null,
"show": true
},
{
"format": "ops",
"label": null,
"logBase": 1,
"max": null,
"min": null,
"show": true
}
],
"yaxis": {
"align": false,
"alignLevel": null
}
}
],
"title": "FuseClient",
"type": "row"
},
{
"collapsed": true,
"gridPos": {
"h": 1,
"w": 24,
"x": 0,
"y": 39
},
"id": 60,
"panels": [
{
"aliasColors": {},
"bars": false,
"dashLength": 10,
"dashes": false,
"datasource": null,
"fill": 1,
"gridPos": {
"h": 6,
"w": 7,
"x": 0,
"y": 2
},
"id": 61,
"legend": {
"avg": false,
"current": false,
"max": false,
"min": false,
"show": true,
"total": false,
"values": false
},
"lines": true,
"linewidth": 1,
"links": [],
"nullPointMode": "null",
"percentage": false,
"pointradius": 5,
"points": false,
"renderer": "flot",
"seriesOverrides": [],
"spaceLength": 10,
"stack": false,
"steppedLine": false,
"targets": [
{
"expr": "go_goroutines{app=\"$app\", cluster=\"$cluster\", role=\"$role\"}",
"format": "time_series",
"intervalFactor": 1,
"legendFormat": "go_goroutines_{{instance}}",
"refId": "A"
},
{
"expr": "go_threads{app=\"$app\", cluster=\"$cluster\", role=\"$role\"}",
"format": "time_series",
"hide": true,
"intervalFactor": 1,
"legendFormat": "go_threads_{{instance}}",
"refId": "B"
}
],
"thresholds": [],
"timeFrom": null,
"timeShift": null,
"title": "go_goroutines",
"tooltip": {
"shared": true,
"sort": 0,
"value_type": "individual"
},
"type": "graph",
"xaxis": {
"buckets": null,
"mode": "time",
"name": null,
"show": true,
"values": []
},
"yaxes": [
{
"format": "locale",
"label": null,
"logBase": 1,
"max": null,
"min": null,
"show": true
},
{
"format": "short",
"label": null,
"logBase": 1,
"max": null,
"min": null,
"show": true
}
],
"yaxis": {
"align": false,
"alignLevel": null
}
},
{
"aliasColors": {},
"bars": false,
"dashLength": 10,
"dashes": false,
"datasource": null,
"fill": 1,
"gridPos": {
"h": 6,
"w": 7,
"x": 7,
"y": 2
},
"id": 169,
"legend": {
"avg": false,
"current": false,
"max": false,
"min": false,
"show": true,
"total": false,
"values": false
},
"lines": true,
"linewidth": 1,
"links": [],
"nullPointMode": "null",
"percentage": false,
"pointradius": 5,
"points": false,
"renderer": "flot",
"seriesOverrides": [],
"spaceLength": 10,
"stack": false,
"steppedLine": false,
"targets": [
{
"expr": "go_goroutines{app=\"$app\", cluster=\"$cluster\", role=\"$role\"}",
"format": "time_series",
"hide": true,
"intervalFactor": 1,
"legendFormat": "go_goroutines_{{instance}}",
"refId": "A"
},
{
"expr": "go_threads{app=\"$app\", cluster=\"$cluster\", role=\"$role\"}",
"format": "time_series",
"hide": false,
"intervalFactor": 1,
"legendFormat": "go_threads_{{instance}}",
"refId": "B"
}
],
"thresholds": [],
"timeFrom": null,
"timeShift": null,
"title": "go_threads",
"tooltip": {
"shared": true,
"sort": 0,
"value_type": "individual"
},
"type": "graph",
"xaxis": {
"buckets": null,
"mode": "time",
"name": null,
"show": true,
"values": []
},
"yaxes": [
{
"format": "locale",
"label": null,
"logBase": 1,
"max": null,
"min": null,
"show": true
},
{
"format": "short",
"label": null,
"logBase": 1,
"max": null,
"min": null,
"show": true
}
],
"yaxis": {
"align": false,
"alignLevel": null
}
},
{
"aliasColors": {},
"bars": false,
"dashLength": 10,
"dashes": false,
"datasource": null,
"fill": 1,
"gridPos": {
"h": 6,
"w": 7,
"x": 14,
"y": 2
},
"id": 63,
"legend": {
"avg": false,
"current": false,
"max": false,
"min": false,
"show": true,
"total": false,
"values": false
},
"lines": true,
"linewidth": 1,
"links": [],
"nullPointMode": "null",
"percentage": false,
"pointradius": 5,
"points": false,
"renderer": "flot",
"seriesOverrides": [
{
"alias": "gc_rate",
"yaxis": 2
}
],
"spaceLength": 10,
"stack": false,
"steppedLine": false,
"targets": [
{
"expr": "go_gc_duration_seconds{instance=~\"$instance\"}",
"format": "time_series",
"intervalFactor": 1,
"legendFormat": "seconds_{{quantile}}",
"refId": "A"
},
{
"expr": "rate(go_gc_duration_seconds_count{instance=~\"$instance\"}[1m])",
"format": "time_series",
"intervalFactor": 1,
"legendFormat": "gc_rate",
"refId": "B"
}
],
"thresholds": [],
"timeFrom": null,
"timeShift": null,
"title": "gc_duration",
"tooltip": {
"shared": true,
"sort": 0,
"value_type": "individual"
},
"type": "graph",
"xaxis": {
"buckets": null,
"mode": "time",
"name": null,
"show": true,
"values": []
},
"yaxes": [
{
"format": "s",
"label": null,
"logBase": 1,
"max": null,
"min": null,
"show": true
},
{
"format": "locale",
"label": null,
"logBase": 1,
"max": null,
"min": null,
"show": true
}
],
"yaxis": {
"align": false,
"alignLevel": null
}
},
{
"aliasColors": {},
"bars": false,
"dashLength": 10,
"dashes": false,
"datasource": null,
"fill": 1,
"gridPos": {
"h": 6,
"w": 7,
"x": 0,
"y": 8
},
"id": 109,
"legend": {
"avg": false,
"current": false,
"max": false,
"min": false,
"show": true,
"total": false,
"values": false
},
"lines": true,
"linewidth": 1,
"links": [],
"nullPointMode": "null",
"percentage": false,
"pointradius": 5,
"points": false,
"renderer": "flot",
"seriesOverrides": [
{
"alias": "gc_rate",
"yaxis": 2
}
],
"spaceLength": 10,
"stack": false,
"steppedLine": false,
"targets": [
{
"expr": "process_resident_memory_bytes{app=\"$app\", cluster=\"$cluster\", role=\"$role\" }",
"format": "time_series",
"hide": false,
"intervalFactor": 1,
"legendFormat": "resident_memory_bytes_{{instance}}",
"refId": "A"
},
{
"expr": "process_virtual_memory_bytes{app=\"$app\", cluster=\"$cluster\", role=\"$role\"}",
"format": "time_series",
"hide": true,
"intervalFactor": 1,
"legendFormat": "virtual_memory_bytes_{{instance}}",
"refId": "B"
},
{
"expr": "process_virtual_memory_max_bytes{app=\"$app\", cluster=\"$cluster\", role=\"$role\"}",
"format": "time_series",
"hide": true,
"intervalFactor": 1,
"legendFormat": "virtual_memory_max_bytes_{{instance}}",
"refId": "C"
}
],
"thresholds": [],
"timeFrom": null,
"timeShift": null,
"title": "process_resident_memory",
"tooltip": {
"shared": true,
"sort": 0,
"value_type": "individual"
},
"type": "graph",
"xaxis": {
"buckets": null,
"mode": "time",
"name": null,
"show": true,
"values": []
},
"yaxes": [
{
"format": "decbytes",
"label": null,
"logBase": 1,
"max": null,
"min": null,
"show": true
},
{
"format": "locale",
"label": null,
"logBase": 1,
"max": null,
"min": null,
"show": true
}
],
"yaxis": {
"align": false,
"alignLevel": null
}
},
{
"aliasColors": {},
"bars": false,
"dashLength": 10,
"dashes": false,
"datasource": null,
"fill": 1,
"gridPos": {
"h": 6,
"w": 7,
"x": 7,
"y": 8
},
"id": 166,
"legend": {
"avg": false,
"current": false,
"max": false,
"min": false,
"show": true,
"total": false,
"values": false
},
"lines": true,
"linewidth": 1,
"links": [],
"nullPointMode": "null",
"percentage": false,
"pointradius": 5,
"points": false,
"renderer": "flot",
"seriesOverrides": [
{
"alias": "gc_rate",
"yaxis": 2
}
],
"spaceLength": 10,
"stack": false,
"steppedLine": false,
"targets": [
{
"expr": "process_resident_memory_bytes{app=\"$app\", cluster=\"$cluster\", role=\"$role\" }",
"format": "time_series",
"hide": true,
"intervalFactor": 1,
"legendFormat": "resident_memory_bytes_{{instance}}",
"refId": "A"
},
{
"expr": "process_virtual_memory_bytes{app=\"$app\", cluster=\"$cluster\", role=\"$role\"}",
"format": "time_series",
"hide": false,
"intervalFactor": 1,
"legendFormat": "virtual_memory_bytes_{{instance}}",
"refId": "B"
},
{
"expr": "process_virtual_memory_max_bytes{app=\"$app\", cluster=\"$cluster\", role=\"$role\"}",
"format": "time_series",
"hide": true,
"intervalFactor": 1,
"legendFormat": "virtual_memory_max_bytes_{{instance}}",
"refId": "C"
}
],
"thresholds": [],
"timeFrom": null,
"timeShift": null,
"title": "process_virtual_memory",
"tooltip": {
"shared": true,
"sort": 0,
"value_type": "individual"
},
"type": "graph",
"xaxis": {
"buckets": null,
"mode": "time",
"name": null,
"show": true,
"values": []
},
"yaxes": [
{
"format": "decbytes",
"label": null,
"logBase": 1,
"max": null,
"min": null,
"show": true
},
{
"format": "locale",
"label": null,
"logBase": 1,
"max": null,
"min": null,
"show": true
}
],
"yaxis": {
"align": false,
"alignLevel": null
}
},
{
"aliasColors": {},
"bars": false,
"dashLength": 10,
"dashes": false,
"datasource": null,
"fill": 1,
"gridPos": {
"h": 6,
"w": 7,
"x": 14,
"y": 8
},
"id": 167,
"legend": {
"avg": false,
"current": false,
"max": false,
"min": false,
"show": true,
"total": false,
"values": false
},
"lines": true,
"linewidth": 1,
"links": [],
"nullPointMode": "null",
"percentage": false,
"pointradius": 5,
"points": false,
"renderer": "flot",
"seriesOverrides": [
{
"alias": "gc_rate",
"yaxis": 2
}
],
"spaceLength": 10,
"stack": false,
"steppedLine": false,
"targets": [
{
"expr": "process_resident_memory_bytes{app=\"$app\", cluster=\"$cluster\", role=\"$role\" }",
"format": "time_series",
"hide": true,
"intervalFactor": 1,
"legendFormat": "resident_memory_bytes_{{instance}}",
"refId": "A"
},
{
"expr": "process_virtual_memory_bytes{app=\"$app\", cluster=\"$cluster\", role=\"$role\"}",
"format": "time_series",
"hide": true,
"intervalFactor": 1,
"legendFormat": "virtual_memory_bytes_{{instance}}",
"refId": "B"
},
{
"expr": "process_virtual_memory_max_bytes{app=\"$app\", cluster=\"$cluster\", role=\"$role\"}",
"format": "time_series",
"hide": false,
"intervalFactor": 1,
"legendFormat": "virtual_memory_max_bytes_{{instance}}",
"refId": "C"
}
],
"thresholds": [],
"timeFrom": null,
"timeShift": null,
"title": "process_virtual_memory_max",
"tooltip": {
"shared": true,
"sort": 0,
"value_type": "individual"
},
"type": "graph",
"xaxis": {
"buckets": null,
"mode": "time",
"name": null,
"show": true,
"values": []
},
"yaxes": [
{
"format": "decbytes",
"label": null,
"logBase": 1,
"max": null,
"min": null,
"show": true
},
{
"format": "locale",
"label": null,
"logBase": 1,
"max": null,
"min": null,
"show": true
}
],
"yaxis": {
"align": false,
"alignLevel": null
}
},
{
"aliasColors": {},
"bars": false,
"dashLength": 10,
"dashes": false,
"datasource": "default",
"fill": 1,
"gridPos": {
"h": 6,
"w": 7,
"x": 0,
"y": 14
},
"id": 110,
"legend": {
"avg": false,
"current": false,
"max": false,
"min": false,
"show": true,
"total": false,
"values": false
},
"lines": true,
"linewidth": 1,
"links": [],
"nullPointMode": "null",
"percentage": false,
"pointradius": 5,
"points": false,
"renderer": "flot",
"seriesOverrides": [
{
"alias": "gc_rate",
"yaxis": 2
}
],
"spaceLength": 10,
"stack": false,
"steppedLine": false,
"targets": [
{
"expr": "process_open_fds{app=\"$app\", cluster=\"$cluster\", role=\"$role\"}",
"format": "time_series",
"intervalFactor": 1,
"legendFormat": "open_fds_{{instance}}",
"refId": "A"
},
{
"expr": "process_max_fds{app=\"$app\", cluster=\"$cluster\", role=\"$role\"}",
"format": "time_series",
"hide": true,
"intervalFactor": 1,
"legendFormat": "max_fds_{{instance}}",
"refId": "B"
}
],
"thresholds": [],
"timeFrom": null,
"timeShift": null,
"title": "open_fds",
"tooltip": {
"shared": true,
"sort": 0,
"value_type": "individual"
},
"type": "graph",
"xaxis": {
"buckets": null,
"mode": "time",
"name": null,
"show": true,
"values": []
},
"yaxes": [
{
"format": "locale",
"label": null,
"logBase": 1,
"max": null,
"min": null,
"show": true
},
{
"format": "locale",
"label": null,
"logBase": 1,
"max": null,
"min": null,
"show": true
}
],
"yaxis": {
"align": false,
"alignLevel": null
}
},
{
"aliasColors": {},
"bars": false,
"dashLength": 10,
"dashes": false,
"datasource": "default",
"fill": 1,
"gridPos": {
"h": 6,
"w": 7,
"x": 7,
"y": 14
},
"id": 168,
"legend": {
"avg": false,
"current": false,
"max": false,
"min": false,
"show": true,
"total": false,
"values": false
},
"lines": true,
"linewidth": 1,
"links": [],
"nullPointMode": "null",
"percentage": false,
"pointradius": 5,
"points": false,
"renderer": "flot",
"seriesOverrides": [
{
"alias": "gc_rate",
"yaxis": 2
}
],
"spaceLength": 10,
"stack": false,
"steppedLine": false,
"targets": [
{
"expr": "process_open_fds{app=\"$app\", cluster=\"$cluster\", role=\"$role\"}",
"format": "time_series",
"hide": true,
"intervalFactor": 1,
"legendFormat": "open_fds_{{instance}}",
"refId": "A"
},
{
"expr": "process_max_fds{app=\"$app\", cluster=\"$cluster\", role=\"$role\"}",
"format": "time_series",
"hide": false,
"intervalFactor": 1,
"legendFormat": "max_fds_{{instance}}",
"refId": "B"
}
],
"thresholds": [],
"timeFrom": null,
"timeShift": null,
"title": "max_fds",
"tooltip": {
"shared": true,
"sort": 0,
"value_type": "individual"
},
"type": "graph",
"xaxis": {
"buckets": null,
"mode": "time",
"name": null,
"show": true,
"values": []
},
"yaxes": [
{
"format": "locale",
"label": null,
"logBase": 1,
"max": null,
"min": null,
"show": true
},
{
"format": "locale",
"label": null,
"logBase": 1,
"max": null,
"min": null,
"show": true
}
],
"yaxis": {
"align": false,
"alignLevel": null
}
},
{
"aliasColors": {},
"bars": false,
"dashLength": 10,
"dashes": false,
"datasource": null,
"fill": 1,
"gridPos": {
"h": 6,
"w": 7,
"x": 14,
"y": 14
},
"id": 111,
"legend": {
"avg": false,
"current": false,
"max": false,
"min": false,
"show": true,
"total": false,
"values": false
},
"lines": true,
"linewidth": 1,
"links": [],
"nullPointMode": "null",
"percentage": false,
"pointradius": 5,
"points": false,
"renderer": "flot",
"seriesOverrides": [
{
"alias": "gc_rate",
"yaxis": 2
}
],
"spaceLength": 10,
"stack": false,
"steppedLine": false,
"targets": [
{
"expr": "rate(process_cpu_seconds_total{cluster=\"$cluster\", role=\"$role\"}[1m])",
"format": "time_series",
"intervalFactor": 1,
"legendFormat": "cpu_seconds_{{instance}}",
"refId": "A"
}
],
"thresholds": [],
"timeFrom": null,
"timeShift": null,
"title": "process_cpu",
"tooltip": {
"shared": true,
"sort": 0,
"value_type": "individual"
},
"type": "graph",
"xaxis": {
"buckets": null,
"mode": "time",
"name": null,
"show": true,
"values": []
},
"yaxes": [
{
"format": "s",
"label": null,
"logBase": 1,
"max": null,
"min": null,
"show": true
},
{
"format": "locale",
"label": null,
"logBase": 1,
"max": null,
"min": null,
"show": true
}
],
"yaxis": {
"align": false,
"alignLevel": null
}
},
{
"aliasColors": {},
"bars": false,
"dashLength": 10,
"dashes": false,
"datasource": null,
"fill": 1,
"gridPos": {
"h": 6,
"w": 7,
"x": 0,
"y": 20
},
"id": 62,
"legend": {
"avg": false,
"current": false,
"max": false,
"min": false,
"show": true,
"total": false,
"values": false
},
"lines": true,
"linewidth": 1,
"links": [],
"nullPointMode": "null",
"percentage": false,
"pointradius": 5,
"points": false,
"renderer": "flot",
"seriesOverrides": [],
"spaceLength": 10,
"stack": false,
"steppedLine": false,
"targets": [
{
"expr": "go_memstats_alloc_bytes{app=\"$app\", cluster=\"$cluster\", role=\"$role\"}",
"format": "time_series",
"intervalFactor": 1,
"legendFormat": "alloc_bytes_{{instance}}",
"refId": "A"
},
{
"expr": "go_memstats_alloc_bytes_total{app=\"$app\", cluster=\"$cluster\", role=\"$role\"}",
"format": "time_series",
"hide": true,
"intervalFactor": 1,
"legendFormat": "alloc_bytes_total_{{instance}}",
"refId": "B"
},
{
"expr": "go_memstats_heap_alloc_bytes{app=\"$app\", cluster=\"$cluster\", role=\"$role\"}",
"format": "time_series",
"hide": true,
"intervalFactor": 1,
"legendFormat": "heap_alloc_bytes_{{instance}}",
"refId": "C"
},
{
"expr": "go_memstats_heap_inuse_bytes{app=\"$app\", cluster=\"$cluster\", role=\"$role\"}",
"format": "time_series",
"hide": true,
"intervalFactor": 1,
"legendFormat": "heap_inuse_bytes_{{instance}}",
"refId": "D"
},
{
"expr": "go_memstats_sys_bytes{app=\"$app\", cluster=\"$cluster\", role=\"$role\"}",
"format": "time_series",
"hide": true,
"intervalFactor": 1,
"legendFormat": "sys_bytes_{{instance}}",
"refId": "E"
},
{
"expr": "go_memstats_mallocs_total{app=\"$app\", cluster=\"$cluster\", role=\"$role\"}",
"format": "time_series",
"hide": true,
"intervalFactor": 1,
"legendFormat": "mallocs_total_{{instance}}",
"refId": "F"
},
{
"expr": "go_memstats_frees_total{app=\"$app\", cluster=\"$cluster\", role=\"$role\"}",
"format": "time_series",
"hide": true,
"intervalFactor": 1,
"legendFormat": "frees_total_{{instance}}",
"refId": "G"
},
{
"expr": "go_memstats_other_sys_bytes{app=\"$app\", cluster=\"$cluster\", role=\"$role\"}",
"format": "time_series",
"hide": true,
"intervalFactor": 1,
"legendFormat": "other_sys_bytes_{{instance}}",
"refId": "H"
},
{
"expr": "go_memstats_mcache_sys_bytes{app=\"$app\", cluster=\"$cluster\", role=\"$role\"}",
"format": "time_series",
"hide": true,
"intervalFactor": 1,
"legendFormat": "mcache_sys_bytes_{{instance}}",
"refId": "I"
}
],
"thresholds": [],
"timeFrom": null,
"timeShift": null,
"title": "alloc_bytes",
"tooltip": {
"shared": true,
"sort": 0,
"value_type": "individual"
},
"type": "graph",
"xaxis": {
"buckets": null,
"mode": "time",
"name": null,
"show": true,
"values": []
},
"yaxes": [
{
"format": "decbytes",
"label": null,
"logBase": 1,
"max": null,
"min": null,
"show": true
},
{
"format": "short",
"label": null,
"logBase": 1,
"max": null,
"min": null,
"show": true
}
],
"yaxis": {
"align": false,
"alignLevel": null
}
},
{
"aliasColors": {},
"bars": false,
"dashLength": 10,
"dashes": false,
"datasource": null,
"fill": 1,
"gridPos": {
"h": 6,
"w": 7,
"x": 7,
"y": 20
},
"id": 172,
"legend": {
"avg": false,
"current": false,
"max": false,
"min": false,
"show": true,
"total": false,
"values": false
},
"lines": true,
"linewidth": 1,
"links": [],
"nullPointMode": "null",
"percentage": false,
"pointradius": 5,
"points": false,
"renderer": "flot",
"seriesOverrides": [],
"spaceLength": 10,
"stack": false,
"steppedLine": false,
"targets": [
{
"expr": "go_memstats_alloc_bytes{app=\"$app\", cluster=\"$cluster\", role=\"$role\"}",
"format": "time_series",
"hide": true,
"intervalFactor": 1,
"legendFormat": "alloc_bytes_{{instance}}",
"refId": "A"
},
{
"expr": "go_memstats_alloc_bytes_total{app=\"$app\", cluster=\"$cluster\", role=\"$role\"}",
"format": "time_series",
"hide": true,
"intervalFactor": 1,
"legendFormat": "alloc_bytes_total_{{instance}}",
"refId": "B"
},
{
"expr": "go_memstats_heap_alloc_bytes{app=\"$app\", cluster=\"$cluster\", role=\"$role\"}",
"format": "time_series",
"hide": false,
"intervalFactor": 1,
"legendFormat": "heap_alloc_bytes_{{instance}}",
"refId": "C"
},
{
"expr": "go_memstats_heap_inuse_bytes{app=\"$app\", cluster=\"$cluster\", role=\"$role\"}",
"format": "time_series",
"hide": true,
"intervalFactor": 1,
"legendFormat": "heap_inuse_bytes_{{instance}}",
"refId": "D"
},
{
"expr": "go_memstats_sys_bytes{app=\"$app\", cluster=\"$cluster\", role=\"$role\"}",
"format": "time_series",
"hide": true,
"intervalFactor": 1,
"legendFormat": "sys_bytes_{{instance}}",
"refId": "E"
},
{
"expr": "go_memstats_mallocs_total{app=\"$app\", cluster=\"$cluster\", role=\"$role\"}",
"format": "time_series",
"hide": true,
"intervalFactor": 1,
"legendFormat": "mallocs_total_{{instance}}",
"refId": "F"
},
{
"expr": "go_memstats_frees_total{app=\"$app\", cluster=\"$cluster\", role=\"$role\"}",
"format": "time_series",
"hide": true,
"intervalFactor": 1,
"legendFormat": "frees_total_{{instance}}",
"refId": "G"
},
{
"expr": "go_memstats_other_sys_bytes{app=\"$app\", cluster=\"$cluster\", role=\"$role\"}",
"format": "time_series",
"hide": true,
"intervalFactor": 1,
"legendFormat": "other_sys_bytes_{{instance}}",
"refId": "H"
},
{
"expr": "go_memstats_mcache_sys_bytes{app=\"$app\", cluster=\"$cluster\", role=\"$role\"}",
"format": "time_series",
"hide": true,
"intervalFactor": 1,
"legendFormat": "mcache_sys_bytes_{{instance}}",
"refId": "I"
}
],
"thresholds": [],
"timeFrom": null,
"timeShift": null,
"title": "heap_alloc",
"tooltip": {
"shared": true,
"sort": 0,
"value_type": "individual"
},
"type": "graph",
"xaxis": {
"buckets": null,
"mode": "time",
"name": null,
"show": true,
"values": []
},
"yaxes": [
{
"format": "decbytes",
"label": null,
"logBase": 1,
"max": null,
"min": null,
"show": true
},
{
"format": "short",
"label": null,
"logBase": 1,
"max": null,
"min": null,
"show": true
}
],
"yaxis": {
"align": false,
"alignLevel": null
}
},
{
"aliasColors": {},
"bars": false,
"dashLength": 10,
"dashes": false,
"datasource": null,
"fill": 1,
"gridPos": {
"h": 6,
"w": 7,
"x": 14,
"y": 20
},
"id": 171,
"legend": {
"avg": false,
"current": false,
"max": false,
"min": false,
"show": true,
"total": false,
"values": false
},
"lines": true,
"linewidth": 1,
"links": [],
"nullPointMode": "null",
"percentage": false,
"pointradius": 5,
"points": false,
"renderer": "flot",
"seriesOverrides": [],
"spaceLength": 10,
"stack": false,
"steppedLine": false,
"targets": [
{
"expr": "go_memstats_alloc_bytes{app=\"$app\", cluster=\"$cluster\", role=\"$role\"}",
"format": "time_series",
"hide": true,
"intervalFactor": 1,
"legendFormat": "alloc_bytes_{{instance}}",
"refId": "A"
},
{
"expr": "go_memstats_alloc_bytes_total{app=\"$app\", cluster=\"$cluster\", role=\"$role\"}",
"format": "time_series",
"hide": true,
"intervalFactor": 1,
"legendFormat": "alloc_bytes_total_{{instance}}",
"refId": "B"
},
{
"expr": "go_memstats_heap_alloc_bytes{app=\"$app\", cluster=\"$cluster\", role=\"$role\"}",
"format": "time_series",
"hide": true,
"intervalFactor": 1,
"legendFormat": "heap_alloc_bytes_{{instance}}",
"refId": "C"
},
{
"expr": "go_memstats_heap_inuse_bytes{app=\"$app\", cluster=\"$cluster\", role=\"$role\"}",
"format": "time_series",
"hide": false,
"intervalFactor": 1,
"legendFormat": "heap_inuse_bytes_{{instance}}",
"refId": "D"
},
{
"expr": "go_memstats_sys_bytes{app=\"$app\", cluster=\"$cluster\", role=\"$role\"}",
"format": "time_series",
"hide": true,
"intervalFactor": 1,
"legendFormat": "sys_bytes_{{instance}}",
"refId": "E"
},
{
"expr": "go_memstats_mallocs_total{app=\"$app\", cluster=\"$cluster\", role=\"$role\"}",
"format": "time_series",
"hide": true,
"intervalFactor": 1,
"legendFormat": "mallocs_total_{{instance}}",
"refId": "F"
},
{
"expr": "go_memstats_frees_total{app=\"$app\", cluster=\"$cluster\", role=\"$role\"}",
"format": "time_series",
"hide": true,
"intervalFactor": 1,
"legendFormat": "frees_total_{{instance}}",
"refId": "G"
},
{
"expr": "go_memstats_other_sys_bytes{app=\"$app\", cluster=\"$cluster\", role=\"$role\"}",
"format": "time_series",
"hide": true,
"intervalFactor": 1,
"legendFormat": "other_sys_bytes_{{instance}}",
"refId": "H"
},
{
"expr": "go_memstats_mcache_sys_bytes{app=\"$app\", cluster=\"$cluster\", role=\"$role\"}",
"format": "time_series",
"hide": true,
"intervalFactor": 1,
"legendFormat": "mcache_sys_bytes_{{instance}}",
"refId": "I"
}
],
"thresholds": [],
"timeFrom": null,
"timeShift": null,
"title": "heap_inuse",
"tooltip": {
"shared": true,
"sort": 0,
"value_type": "individual"
},
"type": "graph",
"xaxis": {
"buckets": null,
"mode": "time",
"name": null,
"show": true,
"values": []
},
"yaxes": [
{
"format": "decbytes",
"label": null,
"logBase": 1,
"max": null,
"min": null,
"show": true
},
{
"format": "short",
"label": null,
"logBase": 1,
"max": null,
"min": null,
"show": true
}
],
"yaxis": {
"align": false,
"alignLevel": null
}
},
{
"aliasColors": {},
"bars": false,
"dashLength": 10,
"dashes": false,
"datasource": null,
"fill": 1,
"gridPos": {
"h": 6,
"w": 7,
"x": 0,
"y": 26
},
"id": 170,
"legend": {
"avg": false,
"current": false,
"max": false,
"min": false,
"show": true,
"total": false,
"values": false
},
"lines": true,
"linewidth": 1,
"links": [],
"nullPointMode": "null",
"percentage": false,
"pointradius": 5,
"points": false,
"renderer": "flot",
"seriesOverrides": [],
"spaceLength": 10,
"stack": false,
"steppedLine": false,
"targets": [
{
"expr": "go_memstats_alloc_bytes{app=\"$app\", cluster=\"$cluster\", role=\"$role\"}",
"format": "time_series",
"hide": true,
"intervalFactor": 1,
"legendFormat": "alloc_bytes_{{instance}}",
"refId": "A"
},
{
"expr": "go_memstats_alloc_bytes_total{app=\"$app\", cluster=\"$cluster\", role=\"$role\"}",
"format": "time_series",
"hide": true,
"intervalFactor": 1,
"legendFormat": "alloc_bytes_total_{{instance}}",
"refId": "B"
},
{
"expr": "go_memstats_heap_alloc_bytes{app=\"$app\", cluster=\"$cluster\", role=\"$role\"}",
"format": "time_series",
"hide": true,
"intervalFactor": 1,
"legendFormat": "heap_alloc_bytes_{{instance}}",
"refId": "C"
},
{
"expr": "go_memstats_heap_inuse_bytes{app=\"$app\", cluster=\"$cluster\", role=\"$role\"}",
"format": "time_series",
"hide": true,
"intervalFactor": 1,
"legendFormat": "heap_inuse_bytes_{{instance}}",
"refId": "D"
},
{
"expr": "go_memstats_sys_bytes{app=\"$app\", cluster=\"$cluster\", role=\"$role\"}",
"format": "time_series",
"hide": false,
"intervalFactor": 1,
"legendFormat": "sys_bytes_{{instance}}",
"refId": "E"
},
{
"expr": "go_memstats_mallocs_total{app=\"$app\", cluster=\"$cluster\", role=\"$role\"}",
"format": "time_series",
"hide": true,
"intervalFactor": 1,
"legendFormat": "mallocs_total_{{instance}}",
"refId": "F"
},
{
"expr": "go_memstats_frees_total{app=\"$app\", cluster=\"$cluster\", role=\"$role\"}",
"format": "time_series",
"hide": true,
"intervalFactor": 1,
"legendFormat": "frees_total_{{instance}}",
"refId": "G"
},
{
"expr": "go_memstats_other_sys_bytes{app=\"$app\", cluster=\"$cluster\", role=\"$role\"}",
"format": "time_series",
"hide": true,
"intervalFactor": 1,
"legendFormat": "other_sys_bytes_{{instance}}",
"refId": "H"
},
{
"expr": "go_memstats_mcache_sys_bytes{app=\"$app\", cluster=\"$cluster\", role=\"$role\"}",
"format": "time_series",
"hide": true,
"intervalFactor": 1,
"legendFormat": "mcache_sys_bytes_{{instance}}",
"refId": "I"
}
],
"thresholds": [],
"timeFrom": null,
"timeShift": null,
"title": "sys_bytes",
"tooltip": {
"shared": true,
"sort": 0,
"value_type": "individual"
},
"type": "graph",
"xaxis": {
"buckets": null,
"mode": "time",
"name": null,
"show": true,
"values": []
},
"yaxes": [
{
"format": "decbytes",
"label": null,
"logBase": 1,
"max": null,
"min": null,
"show": true
},
{
"format": "short",
"label": null,
"logBase": 1,
"max": null,
"min": null,
"show": true
}
],
"yaxis": {
"align": false,
"alignLevel": null
}
},
{
"aliasColors": {},
"bars": false,
"dashLength": 10,
"dashes": false,
"datasource": null,
"fill": 1,
"gridPos": {
"h": 6,
"w": 7,
"x": 7,
"y": 26
},
"id": 173,
"legend": {
"avg": false,
"current": false,
"max": false,
"min": false,
"show": true,
"total": false,
"values": false
},
"lines": true,
"linewidth": 1,
"links": [],
"nullPointMode": "null",
"percentage": false,
"pointradius": 5,
"points": false,
"renderer": "flot",
"seriesOverrides": [],
"spaceLength": 10,
"stack": false,
"steppedLine": false,
"targets": [
{
"expr": "go_memstats_alloc_bytes{app=\"$app\", cluster=\"$cluster\", role=\"$role\"}",
"format": "time_series",
"hide": true,
"intervalFactor": 1,
"legendFormat": "alloc_bytes_{{instance}}",
"refId": "A"
},
{
"expr": "go_memstats_alloc_bytes_total{app=\"$app\", cluster=\"$cluster\", role=\"$role\"}",
"format": "time_series",
"hide": true,
"intervalFactor": 1,
"legendFormat": "alloc_bytes_total_{{instance}}",
"refId": "B"
},
{
"expr": "go_memstats_heap_alloc_bytes{app=\"$app\", cluster=\"$cluster\", role=\"$role\"}",
"format": "time_series",
"hide": true,
"intervalFactor": 1,
"legendFormat": "heap_alloc_bytes_{{instance}}",
"refId": "C"
},
{
"expr": "go_memstats_heap_inuse_bytes{app=\"$app\", cluster=\"$cluster\", role=\"$role\"}",
"format": "time_series",
"hide": true,
"intervalFactor": 1,
"legendFormat": "heap_inuse_bytes_{{instance}}",
"refId": "D"
},
{
"expr": "go_memstats_sys_bytes{app=\"$app\", cluster=\"$cluster\", role=\"$role\"}",
"format": "time_series",
"hide": true,
"intervalFactor": 1,
"legendFormat": "sys_bytes_{{instance}}",
"refId": "E"
},
{
"expr": "go_memstats_mallocs_total{app=\"$app\", cluster=\"$cluster\", role=\"$role\"}",
"format": "time_series",
"hide": true,
"intervalFactor": 1,
"legendFormat": "mallocs_total_{{instance}}",
"refId": "F"
},
{
"expr": "go_memstats_frees_total{app=\"$app\", cluster=\"$cluster\", role=\"$role\"}",
"format": "time_series",
"hide": true,
"intervalFactor": 1,
"legendFormat": "frees_total_{{instance}}",
"refId": "G"
},
{
"expr": "go_memstats_other_sys_bytes{app=\"$app\", cluster=\"$cluster\", role=\"$role\"}",
"format": "time_series",
"hide": false,
"intervalFactor": 1,
"legendFormat": "other_sys_bytes_{{instance}}",
"refId": "H"
},
{
"expr": "go_memstats_mcache_sys_bytes{app=\"$app\", cluster=\"$cluster\", role=\"$role\"}",
"format": "time_series",
"hide": true,
"intervalFactor": 1,
"legendFormat": "mcache_sys_bytes_{{instance}}",
"refId": "I"
}
],
"thresholds": [],
"timeFrom": null,
"timeShift": null,
"title": "other_sys",
"tooltip": {
"shared": true,
"sort": 0,
"value_type": "individual"
},
"type": "graph",
"xaxis": {
"buckets": null,
"mode": "time",
"name": null,
"show": true,
"values": []
},
"yaxes": [
{
"format": "decbytes",
"label": null,
"logBase": 1,
"max": null,
"min": null,
"show": true
},
{
"format": "short",
"label": null,
"logBase": 1,
"max": null,
"min": null,
"show": true
}
],
"yaxis": {
"align": false,
"alignLevel": null
}
},
{
"aliasColors": {},
"bars": false,
"dashLength": 10,
"dashes": false,
"datasource": null,
"fill": 1,
"gridPos": {
"h": 6,
"w": 7,
"x": 14,
"y": 26
},
"id": 174,
"legend": {
"avg": false,
"current": false,
"max": false,
"min": false,
"show": true,
"total": false,
"values": false
},
"lines": true,
"linewidth": 1,
"links": [],
"nullPointMode": "null",
"percentage": false,
"pointradius": 5,
"points": false,
"renderer": "flot",
"seriesOverrides": [],
"spaceLength": 10,
"stack": false,
"steppedLine": false,
"targets": [
{
"expr": "go_memstats_alloc_bytes{app=\"$app\", cluster=\"$cluster\", role=\"$role\"}",
"format": "time_series",
"hide": true,
"intervalFactor": 1,
"legendFormat": "alloc_bytes_{{instance}}",
"refId": "A"
},
{
"expr": "go_memstats_alloc_bytes_total{app=\"$app\", cluster=\"$cluster\", role=\"$role\"}",
"format": "time_series",
"hide": true,
"intervalFactor": 1,
"legendFormat": "alloc_bytes_total_{{instance}}",
"refId": "B"
},
{
"expr": "go_memstats_heap_alloc_bytes{app=\"$app\", cluster=\"$cluster\", role=\"$role\"}",
"format": "time_series",
"hide": true,
"intervalFactor": 1,
"legendFormat": "heap_alloc_bytes_{{instance}}",
"refId": "C"
},
{
"expr": "go_memstats_heap_inuse_bytes{app=\"$app\", cluster=\"$cluster\", role=\"$role\"}",
"format": "time_series",
"hide": true,
"intervalFactor": 1,
"legendFormat": "heap_inuse_bytes_{{instance}}",
"refId": "D"
},
{
"expr": "go_memstats_sys_bytes{app=\"$app\", cluster=\"$cluster\", role=\"$role\"}",
"format": "time_series",
"hide": true,
"intervalFactor": 1,
"legendFormat": "sys_bytes_{{instance}}",
"refId": "E"
},
{
"expr": "go_memstats_mallocs_total{app=\"$app\", cluster=\"$cluster\", role=\"$role\"}",
"format": "time_series",
"hide": true,
"intervalFactor": 1,
"legendFormat": "mallocs_total_{{instance}}",
"refId": "F"
},
{
"expr": "go_memstats_frees_total{app=\"$app\", cluster=\"$cluster\", role=\"$role\"}",
"format": "time_series",
"hide": true,
"intervalFactor": 1,
"legendFormat": "frees_total_{{instance}}",
"refId": "G"
},
{
"expr": "go_memstats_other_sys_bytes{app=\"$app\", cluster=\"$cluster\", role=\"$role\"}",
"format": "time_series",
"hide": true,
"intervalFactor": 1,
"legendFormat": "other_sys_bytes_{{instance}}",
"refId": "H"
},
{
"expr": "go_memstats_mcache_sys_bytes{app=\"$app\", cluster=\"$cluster\", role=\"$role\"}",
"format": "time_series",
"hide": false,
"intervalFactor": 1,
"legendFormat": "mcache_sys_bytes_{{instance}}",
"refId": "I"
}
],
"thresholds": [],
"timeFrom": null,
"timeShift": null,
"title": "mcache_sys",
"tooltip": {
"shared": true,
"sort": 0,
"value_type": "individual"
},
"type": "graph",
"xaxis": {
"buckets": null,
"mode": "time",
"name": null,
"show": true,
"values": []
},
"yaxes": [
{
"format": "decbytes",
"label": null,
"logBase": 1,
"max": null,
"min": null,
"show": true
},
{
"format": "short",
"label": null,
"logBase": 1,
"max": null,
"min": null,
"show": true
}
],
"yaxis": {
"align": false,
"alignLevel": null
}
}
],
"title": "GoRuntime",
"type": "row"
}
],
"refresh": false,
"schemaVersion": 16,
"style": "dark",
"tags": [],
"templating": {
"list": [
{
"allValue": null,
"current": {
"selected": true,
"text": "cfs",
"value": "cfs"
},
"hide": 2,
"includeAll": false,
"label": "App",
"multi": false,
"name": "app",
"options": [
{
"selected": true,
"text": "cfs",
"value": "cfs"
}
],
"query": "cfs",
"type": "custom"
},
{
"allValue": null,
"current": {
"selected": false,
"text": "click_stream",
"value": "click_stream"
},
"datasource": "default",
"hide": 0,
"includeAll": false,
"label": "Cluster",
"multi": false,
"name": "cluster",
"options": [],
"query": "label_values(go_info{app=~\"$app\"}, cluster)",
"refresh": 1,
"regex": "",
"sort": 0,
"tagValuesQuery": "",
"tags": [],
"tagsQuery": "",
"type": "query",
"useTags": false
},
{
"allValue": null,
"current": {
"selected": false,
"text": "master",
"value": "master"
},
"datasource": "default",
"hide": 0,
"includeAll": false,
"label": "Role",
"multi": false,
"name": "role",
"options": [],
"query": "label_values(go_info{app=~\"$app\", cluster=~\"$cluster\"}, role)",
"refresh": 1,
"regex": "",
"sort": 0,
"tagValuesQuery": "",
"tags": [],
"tagsQuery": "",
"type": "query",
"useTags": false
},
{
"allValue": null,
"current": {
"selected": true,
"text": "10.194.138.195:9061",
"value": "10.194.138.195:9061"
},
"datasource": "default",
"hide": 0,
"includeAll": false,
"label": "Instance",
"multi": false,
"name": "instance",
"options": [],
"query": "label_values(go_info{app=~\"$app\", role=~\"$role\", cluster=~\"$cluster\"}, instance)",
"refresh": 1,
"regex": "",
"sort": 0,
"tagValuesQuery": "",
"tags": [],
"tagsQuery": "",
"type": "query",
"useTags": false
},
{
"allValue": null,
"current": {
"selected": true,
"text": "10.194.138.195",
"value": "10.194.138.195"
},
"datasource": "default",
"hide": 2,
"includeAll": false,
"label": "Host",
"multi": false,
"name": "hostip",
"options": [],
"query": "label_values(go_info{instance=~\"$instance\", cluster=~\"$cluster\"}, instance)",
"refresh": 1,
"regex": "/([^:]+):.*/",
"sort": 0,
"tagValuesQuery": "",
"tags": [],
"tagsQuery": "",
"type": "query",
"useTags": false
},
{
"allValue": null,
"current": {
"selected": true,
"text": "zhytest",
"value": "zhytest"
},
"datasource": "default",
"hide": 0,
"includeAll": false,
"label": "Vol",
"multi": false,
"name": "vol",
"options": [],
"query": "label_values(cfs_master_vol_total_GB{app=\"$app\",cluster=\"$cluster\"}, volName)",
"refresh": 1,
"regex": "",
"sort": 0,
"tagValuesQuery": "",
"tags": [],
"tagsQuery": "",
"type": "query",
"useTags": false
}
]
},
"time": {
"from": "now-1h",
"to": "now"
},
"timepicker": {
"refresh_intervals": [
"5s",
"10s",
"30s",
"1m",
"5m",
"15m",
"30m",
"1h",
"2h",
"1d"
],
"time_options": [
"5m",
"15m",
"1h",
"6h",
"12h",
"24h",
"2d",
"7d",
"30d"
]
},
"timezone": "",
"title": "chubaofs",
"uid": "J8XJyOmZk",
"version": 367
}
Tune FUSE Performance¶
Fetch Linux kernel source code¶
Download the corresponding src rpm, and use the following commands to install source code.
rpm -i kernel-3.10.0-327.28.3.el7.src.rpm 2>&1 | grep -v exist
cd ~/rpmbuild/SPECS
rpmbuild -bp --target=$(uname -m) kernel.spec
The source code will be installed in ~/rpmbuild/BUILD/
Optimize FUSE linux kernel module¶
In order to achieve maximum throughput performance, several FUSE kernel parameters have to be modified, such as FUSE_MAX_PAGES_PER_REQ and FUSE_DEFAULT_MAX_BACKGROUND.
Update source code according to the following lines.
/* fs/fuse/fuse_i.h */
#define FUSE_MAX_PAGES_PER_REQ 256
/* fs/fuse/inode.c */
#define FUSE_DEFAULT_MAX_BACKGROUND 32
Build against current running Linux kernel¶
yum install kernel-devel-3.10.0-327.28.3.el7.x86_64
cd ~/rpmbuild/BUILD/kernel-3.10.0-327.28.3.el7/linux-3.10.0-327.28.3.el7.x86_64/fs/fuse
make -C /lib/modules/`uname -r`/build M=$PWD
Install kernel module¶
cp fuse.ko /lib/modules/`uname -r`/kernel/fs/fuse
rmmod fuse
depmod -a
modprobe fuse
Docker¶
Under the docker directory, a helper tool called run_docker.sh is provided to run ChubaoFS with docker-compose.
To start a minimal ChubaoFS cluster from scratch, note that /data/disk is arbitrary, and make sure there are at least 30G available space.
$ docker/run_docker.sh -r -d /data/disk
If client starts successfully, use mount command in client docker shell to check mount status:
$ mount | grep chubaofs
Open http://127.0.0.1:3000 in browser, login with admin/123456 to view grafana monitor metrics.
Or run server and client seperately by following commands:
$ docker/run_docker.sh -b
$ docker/run_docker.sh -s -d /data/disk
$ docker/run_docker.sh -c
$ docker/run_docker.sh -m
For more usage:
$ docker/run_docker.sh -h
Prometheus and Grafana confg can be found in docker/monitor directory.
Resource Manager (Master) API¶
Cluster¶
Overview¶
curl -v "http://127.0.0.1/admin/getCluster" | python -m json.tool
display the base information of the cluster, such as the detail of metaNode,dataNode,vol and so on.
response
{
"Name": "test",
"LeaderAddr": "127.0.0.1:80",
"DisableAutoAlloc": false,
"Applied": 225,
"MaxDataPartitionID": 100,
"MaxMetaNodeID": 3,
"MaxMetaPartitionID": 1,
"DataNodeStat": {},
"MetaNodeStat": {},
"VolStat": {},
"MetaNodes": {},
"DataNodes": {}
}
Volume¶
Create¶
curl -v "http://127.0.0.1/admin/createVol?name=test&capacity=100&owner=cfs&mpCount=3"
Allocate a set of data partition and a meta partition to the user.
Default create 10 data partition and 3 meta partition when create volume.
| Parameter | Type | Description |
|---|---|---|
| name | string | |
| capacity | int | the quota of vol,unit is GB |
| owner | string | the owner of vol |
| mpCount | int | the amount of initial meta partitions |
Delete¶
curl -v "http://127.0.0.1/vol/delete?name=test&authKey=md5(owner)"
Mark the vol status to MarkDelete first, then delete data partition and meta partition asynchronous,finally delete meta data from persist store
| Parameter | Type | Description |
|---|---|---|
| name | string | |
| authKey | string | calculates the MD5 value of the owner field as authentication information |
Get¶
curl -v "http://127.0.0.1/client/vol?name=test&authKey=md5(owner)" | python -m json.tool
show the base information of the vol,such as name,the detail of data partitions and meta partitions and so on.
| Parameter | Type | Description |
|---|---|---|
| name | string | |
| authKey | string | calculates the MD5 value of the owner field as authentication information |
response
{
"Name": "test",
"VolType": "extent",
"MetaPartitions": {},
"DataPartitions": {},
"OSSSecure": {
"AccessKey": "l9aC5S9OzlneHguZ",
"SecretKey": "4sL0xJfHVVHKOC3MDwPfaF1AcGm5a2XL"
}
}
Meta Partition¶
Create¶
curl -v "http://127.0.0.1/metaPartition/create?name=test&start=10000"
split meta partition manually,if max meta partition of the vol which range is [0,end),end larger than start parameter,old meta partition range is[0,start], new meta partition is [start+1,end)
| Parameter | Type | Description |
|---|---|---|
| name | string | the name of vol |
| start | uint64 | the start value of meta partition which will be create |
Get¶
curl -v "http://127.0.0.1/client/metaPartition?id=1" | python -m json.tool
show base information of meta partition,such as id,start,end and so on.
| Parameter | Type | Description |
|---|---|---|
| id | uint64 | the id of meta partition |
response
{
"PartitionID": 1,
"Start": 0,
"End": 9223372036854776000,
"MaxNodeID": 1,
"Replicas": {},
"ReplicaNum": 3,
"Status": 2,
"PersistenceHosts": {},
"Peers": {},
"MissNodes": {}
}
Data Partition¶
Create¶
curl -v "http://127.0.0.1/dataPartition/create?count=400&name=test"
create a set of data partition
| Parameter | Type | Description |
|---|---|---|
| count | int | the num of dataPartitions will be create |
| name | string | the name of vol |
Get¶
curl -v "http://127.0.0.1/dataPartition/get?id=100" | python -m json.tool
| Parameter | Type | Description |
|---|---|---|
| id | uint64 | the id of data partition |
response
{
"PartitionID": 100,
"LastLoadTime": 1544082851,
"ReplicaNum": 3,
"Status": 2,
"Replicas": {},
"PartitionType": "extent",
"PersistenceHosts": {},
"Peers": {},
"MissNodes": {},
"VolName": "test",
"RandomWrite": true,
"FileInCoreMap": {}
}
Meta Node API¶
Meta Partition¶
Get Partitions¶
curl -v http://127.0.0.1:9092/getPartitions
Get all meta-partition base information of the metanode.
Inode¶
Get Inode¶
curl -v http://127.0.0.1:9092/getInode?pid=100&ino=1024
Get inode information
| Parameter | Type | Description |
|---|---|---|
| pid | integer | meta-partition id |
| ino | integer | inode id |
Dentry¶
Get Dentry¶
curl -v 'http://127.0.0.1:9092/getDentry?pid=100&name=""&parentIno=1024'
Get dentry information
| Parameter | Type | Description |
|---|---|---|
| pid | integer | meta partition id |
| name | string | file or directory name |
| parentIno | integer | file or directory parent directory inode |
Use Cases¶
ChubaoFS is a distributed file system that is compatible with most POSIX file system semantics. When ChubaoFS is mounted, it can be as simple as using a local file system. Basically, it can be used in any case where a file system is needed, replacing the local file system, and realizing infinitely expandable storage without physical boundaries. It has been applied in various scenarios, and the following are some of the extracted scenes.
Machine Learning¶
Disadvantages of using a local disk to store training data sets:
- The local disk space is small, and there are multiple models. The training data set of each model reaches the TB level. If you use a local disk to store the training data set, you need to reduce the size of the training data set.
- Training data sets need to be updated frequently, requiring more disk space.
- Risk of loss of training data set if machine crashes.
The advantages of using ChubaoFS to store training data sets:
- Unlimited disk space, easy to expand capacity. It can automatically expand disk capacity according to the percentage of disk usage, enabling storage system capacity expansion on demand, greatly saving storage costs.
- Multiple replicas of data to ensure high data reliability without worrying about losing data.
- Compatible with POSIX file system interface, no changes required by the application.
ElasticSearch¶
Using local disks to store data often encounters the following problems:
- Disk usage is uneven and disk IO cannot be fully utilized.
- Local disk space is limited.
The advantages of using ChubaoFS as a backend storage:
- Unlimited disk space, easy to expand capacity. It can automatically expand disk capacity according to the percentage of disk usage, enabling storage system capacity expansion on demand, greatly saving storage costs.
- Disk IO usage is uniform and disk IO is fully utilized.
- Ensure data is highly reliable without worrying about losing data.
Nginx Log Storage¶
Do you often worry about running out of local disk space? With ChubaoFS, you can store datas in a distributed file system without worrying about running out of disk space. If you use a local disk to store logs, you may often worry about the following issues:
- Docker local disk space is small.
- If the docker container crashes, the log is lost and unrecoverable.
- The mixed deployment of physical machine and docker machine is difficult to manage and has high operation and maintenance cost.
The advantages of using ChubaoFS to store nginx logs are:
- The disk space is unlimited and easy to expand. The disk capacity is automatically expanded according to the percentage of disk usage. The storage system can be expanded on demand, which greatly saves storage costs.
- Ensure data is highly reliable and do not worry about losing data.
- Multiple replicas to solve the problem of unable to write to the log caused by disk error and datanode crashes.
- Compatible with the POSIX file system interface, no changes required by the application.
- The operation and maintenance of ChubaoFS are simple, and one person can easily manage the cluster of tens of thousands of machines.
Spark¶
In the big data set scenario, are you worried about the amount of data stored in Spark intermediate calculation results in order to carefully calculate each task? You can store the shuffle results to ChubaoFS, and no longer worry about the disk has no free space which causes the task to fail. This enables the separation of storage and computation. The pain points of using local disk to store shuffle intermediate results:
- Insufficient disk space.
- Too many temporary directory files to create new files.
The advantages of using ChubaoFS:
- Unlimited disk space, easy to expand capacity. It can automatically expand disk capacity according to the percentage of disk usage, enabling storage system capacity expansion on demand, greatly saving storage costs.
- Meta node manages file metadata, which can be expanded horizontally and the number of files is unlimited.
MySQL Database Backup¶
Disadvantages of using OSS(Object Storage Service) to back up MySQL database:
- Need to use the OSS SDK or RESTful API to develop backup programs which increases the difficulty of operation and maintenance.
- If backup file fails, troubleshooting is more difficult.
- After backing up files to OSS, it is not convenient to check whether the files are successfully uploaded.
- Backup files are processed by multiple layers of services, which affects performance.
Advantages of using ChubaoFS to back up MySQL databases:
- Easy to use, compatible with POSIX file interface, and can be used as a local file system.
- Complete and detailed operation logs are stored in the local file system, making it easy to troubleshoot problems.
- Simply execute the ls command to verify that the file was successfully uploaded.
- Supports PageCache and WriteCache, and file read and write performance is significantly improved compared to OSS.
Performance¶
Environment¶
| Instance | Nodes | CPU | Memory | Storage | Network | Description |
| Master | 3 | 32 | 32 GB | 120 GB SSD | 10 Gb/s | |
| MetaNode | 10 | 32 | 32 GB | 16 x 1TB SSD | 10 Gb/s | hybrid deployment |
| DataNode | 10 | 32 | 32 GB | 16 x 1TB SSD | 10 Gb/s | hybrid deployment |
IO Performance and Scalability¶
IO Performance and benchmark scalability test by fio.
1. Sequential Read¶
Bandwidth
| 1 Process | 4 Processes | 16 Processes | 64 Processes | |
| 1 Client | 148.000 | 626.000 | 1129.000 | 1130.000 |
| 2 Clients | 284.000 | 1241.000 | 2258.000 | 2260.000 |
| 4 Clients | 619.000 | 2640.000 | 4517.000 | 4515.000 |
| 8 Clients | 1193.000 | 4994.000 | 9006.000 | 9034.000 |
IOPS
| 1 Process | 4 Processes | 16 Processes | 64 Processes | |
| 1 Client | 1180.000 | 5007.000 | 9031.000 | 9040.000 |
| 2 Clients | 2275.000 | 9924.000 | 18062.000 | 18081.000 |
| 4 Clients | 4954.000 | 21117.000 | 36129.000 | 36112.000 |
| 8 Clients | 9531.000 | 39954.000 | 72048.000 | 72264.000 |
Latency
| 1 Process | 4 Processes | 16 Processes | 64 Processes | |
| 1 Client | 842.200 | 794.340 | 1767.310 | 7074.550 |
| 2 Clients | 874.255 | 801.690 | 1767.370 | 7071.715 |
| 4 Clients | 812.363 | 760.702 | 1767.710 | 7077.065 |
| 8 Clients | 837.707 | 799.851 | 1772.620 | 7076.967 |
2. Sequential Write¶
Bandwidth
| 1 Process | 4 Processes | 16 Processes | 64 Processes | |
| 1 Client | 52.200 | 226.000 | 956.000 | 1126.000 |
| 2 Clients | 104.500 | 473.000 | 1763.000 | 2252.000 |
| 4 Clients | 225.300 | 1015.000 | 2652.000 | 3472.000 |
| 8 Clients | 480.600 | 1753.000 | 3235.000 | 3608.000 |
IOPS
| 1 Process | 4 Processes | 16 Processes | 64 Processes | |
| 1 Client | 417 | 1805 | 7651 | 9004 |
| 2 Clients | 835 | 3779 | 14103 | 18014 |
| 4 Clients | 1801 | 8127 | 21216 | 27777 |
| 8 Clients | 3841 | 14016 | 25890 | 28860 |
Latency
| 1 Process | 4 Processes | 16 Processes | 64 Processes | |
| 1 Client | 2385.400 | 2190.210 | 2052.360 | 7081.320 |
| 2 Clients | 2383.610 | 2081.850 | 2233.790 | 7079.450 |
| 4 Clients | 2216.305 | 1947.688 | 2946.017 | 8842.903 |
| 8 Clients | 2073.921 | 2256.120 | 4787.496 | 17002.425 |
3. Random Read¶
Bandwidth
| 1 Process | 4 Processes | 16 Processes | 64 Processes | |
| 1 Client | 6.412 | 39.100 | 216.000 | 534.000 |
| 2 Clients | 14.525 | 88.100 | 409.000 | 1002.000 |
| 4 Clients | 33.242 | 200.200 | 705.000 | 1693.000 |
| 8 Clients | 59.480 | 328.300 | 940.000 | 2369.000 |
IOPS
| 1 Process | 4 Processes | 16 Processes | 64 Processes | |
| 1 Client | 1641 | 10240 | 56524.800 | 140288 |
| 2 Clients | 3718 | 23142.4 | 107212.8 | 263168 |
| 4 Clients | 8508 | 52428.8 | 184627.2 | 443392 |
| 8 Clients | 15222 | 85072.8 | 246681.6 | 621056 |
Latency
| 1 Process | 4 Processes | 16 Processes | 64 Processes | |
| 1 Client | 603.580 | 395.420 | 287.510 | 466.320 |
| 2 Clients | 532.840 | 351.815 | 303.460 | 497.100 |
| 4 Clients | 469.025 | 317.140 | 355.105 | 588.847 |
| 8 Clients | 524.709 | 382.862 | 530.811 | 841.985 |
4. Random Write¶
Bandwidth
| 1 Process | 4 Processes | 16 Processes | 64 Processes | |
| 1 Client | 3.620 | 17.500 | 118.000 | 318.000 |
| 2 Clients | 7.540 | 44.800 | 230.000 | 476.000 |
| 4 Clients | 16.245 | 107.700 | 397.900 | 636.000 |
| 8 Clients | 39.274 | 208.100 | 487.100 | 787.100 |
IOPS
| 1 Process | 4 Processes | 16 Processes | 64 Processes | |
| 1 Client | 926.000 | 4476.000 | 31027.200 | 83251.200 |
| 2 Clients | 1929.000 | 11473.000 | 60313.600 | 124620.800 |
| 4 Clients | 4156.000 | 27800.000 | 104243.200 | 167014.400 |
| 8 Clients | 10050.000 | 53250.000 | 127692.800 | 206745.600 |
Latency
| 1 Process | 4 Processes | 16 Processes | 64 Processes | |
| 1 Client | 1073.150 | 887.570 | 523.820 | 784.030 |
| 2 Clients | 1030.010 | 691.530 | 539.525 | 1042.685 |
| 4 Clients | 955.972 | 575.183 | 618.445 | 1552.205 |
| 8 Clients | 789.883 | 598.393 | 1016.185 | 2506.424 |
Metadata Performance and Scalability¶
Metadata performance and scalability benchmark test by mdtest.
Dir Creation
| 1 Process | 4 Processes | 16 Processes | 64 Processes | |
| 1 Client | 448.618 | 2421.001 | 14597.97 | 43055.15 |
| 2 Clients | 956.947 | 5917.576 | 28930.431 | 72388.765 |
| 4 Clients | 2027.02 | 13213.403 | 54449.056 | 104771.356 |
| 8 Clients | 4643.755 | 27416.904 | 89641.301 | 119542.62 |
Dir Removal
| 1 Process | 4 Processes | 16 Processes | 64 Processes | |
| 1 Client | 399.779 | 2118.005 | 12351.635 | 34903.672 |
| 2 Clients | 833.353 | 5176.812 | 24471.674 | 50242.973 |
| 4 Clients | 1853.617 | 11462.927 | 46413.313 | 91128.059 |
| 8 Clients | 4441.435 | 24133.617 | 74401.336 | 115013.557 |
Dir Stat
| 1 Process | 4 Processes | 16 Processes | 64 Processes | |
| 1 Client | 283232.761 | 1215309.524 | 4231088.104 | 12579177.02 |
| 2 Clients | 572834.143 | 2169669.058 | 8362749.217 | 18120970.71 |
| 4 Clients | 1263474.549 | 3333746.786 | 10160929.29 | 31874265.88 |
| 8 Clients | 2258670.069 | 8715752.83 | 22524794.98 | 77533648.04 |
File Creation
| 1 Process | 4 Processes | 16 Processes | 64 Processes | |
| 1 Client | 448.888 | 2400.803 | 13638.072 | 27785.947 |
| 2 Clients | 925.68 | 5664.166 | 25889.163 | 50434.484 |
| 4 Clients | 2001.137 | 12986.968 | 50330.952 | 91387.825 |
| 8 Clients | 4479.831 | 25933.437 | 86667.966 | 112746.199 |
File Removal
| 1 Process | 4 Processes | 16 Processes | 64 Processes | |
| 1 Client | 605.143 | 3678.138 | 18631.342 | 47035.912 |
| 2 Clients | 1301.151 | 8365.667 | 34005.25 | 64860.041 |
| 4 Clients | 3032.683 | 14017.426 | 50938.926 | 80692.761 |
| 8 Clients | 7170.386 | 32056.959 | 68761.908 | 88357.563 |
Tree Creation
| 1 Process | 4 Processes | 16 Processes | 64 Processes | |
| 1 Client | 305.778 | 229.562 | 86.299 | 23.917 |
| 2 Clients | 161.31 | 211.119 | 76.301 | 24.439 |
| 4 Clients | 260.562 | 223.153 | 81.209 | 23.867 |
| 8 Clients | 350.038 | 220.744 | 81.621 | 17.144 |
Tree Removal
| 1 Process | 4 Processes | 16 Processes | 64 Processes | |
| 1 Client | 137.462 | 70.881 | 31.235 | 7.057 |
| 2 Clients | 217.026 | 113.36 | 23.971 | 7.128 |
| 4 Clients | 231.622 | 113.539 | 30.626 | 7.299 |
| 8 Clients | 185.156 | 83.923 | 20.607 | 5.515 |
Integrity¶
- Linux Test Project / fs
Workload¶
database backup
Java application logs
code git repo
database systems
MyRocks, MySQL Innodb, HBase,
Scalability¶
- volume scalability: tens to millions of cfs volumes
- metadata scalability: a big volume with billions of files/directories
FAQ¶
How does ChubaoFS compare with its alternatives ?¶
Ceph¶
Ceph (https://ceph.com/) is a widely-used free-software storage platform. It can be configured for object storage, block storage, as well as file system. But many nice features provided by Ceph (e.g., various storage backends) also make it very complicated and difficult to learn and deploy. In addition, in certain scenarios such as metadata operations and small file operations, its performance in a multi-client environment can be hard to optimize.
GFS and HDFS¶
GFS and its open source implementation HDFS (https://github.com/apache/hadoop) are designed for storing large files with sequential access. Both of them adopt the master-slave architecture, where the single master stores all the file metadata. Unlike GFS and HDFS, ChubaoFS employs a separate metadata subsystem to provide a scalable solution for the metadata storage so that the resource manager has less chance to become the bottleneck.
Haystack¶
Haystack from Facebook takes after log-structured filesystems to serve long tail of requests seen by sharing photos in a large social network. The key insight is to avoid disk operations when accessing metadata. ChubaoFS adopts similar ideas by putting the file metadata into the main memory.
However, different from Haystack, the actually physical offsets instead of logical indices of the file contents are stored in the memory, and deleting a file is achieved by the punch hole interface provided by the underlying file system instead of relying on the garbage collector to perform merging and compacting regularly for more efficient disk utilization. In addition, Haystack does not guarantee the strong consistency among the replicas when deleting the files, and it needs to perform merging and compacting regularly for more efficient disk utilization, which could be a performance killer. ChubaoFS takes a different design principle to separate the storage of file metadata and contents. In this way, we can have more flexible and cost-effective deployments of meta and data nodes.
Public Cloud¶
Windows Azure Storage (https://azure.microsoft.com/en-us/) is a cloud storage system that provides strong consistency and multi-tenancy to the clients. Different from ChubaoFS, it builds an extra partition layer to handle random writes before streaming data into the lower level. AWS EFS (https://aws.amazon.com/efs/) is a cloud storage service that provides scalable and elastic file storage. Depending on the use cases, there could be a considerable amount of cost associated with using these cloud storage services.
Others¶
GlusterFS (https://www.gluster.org/) is a scalable distributed file system that aggregates disk storage resources from multiple servers into a single global namespace. MooseFS (https://moosefs.com/) is a fault- tolerant, highly available, POSIX-compliant, and scalable distributed file system. However, similar to HDFS, it employs a single master to manage the file metadata.