Aliyun Log CLI

User Guide

中文版README

Content

• Introduction
  • Brief
  • Major Features
• Installation
  • Operation System
  • Supported Version
  • Installation Method
  • Offline Installation
  • FAQ of Installation
  • Full Usage list
• Configure CLI
• Input and Output
  • Inputs
  • Parameter Validation
  • Output
  • Filter output
  • Further Process
• Command Reference
  • Command Specification
  • Alias
  • Subcommand and parameters
  • Global options
  • Command categories
    • Project management
    • Logstore management
    • Shard management
    • Machine group management
    • Logtail config management
    • Machine group and Logtail Config Mapping
    • Index management
    • Cursor management
    • Logs write and consume
    • Shipper management
    • Consumer group management
    • Elasticsearch data migration
• Best Practice
• Troubleshooting
• Other resources

Introduction

The Alicloud log service provides with Web and SDK flavor to operate log service and analyzie logs. To make it more convinient to do automation, we release this command line interface (CLI).

Brief

Alicloud log service command line console, support almost all operations as web. It also supports incomplete log query check and query cross multiple pages. It could even do project settings copy cross multiple regions.

Major Features

• Support almost all 50+ REST API of log service.
• Multiple account support to support cross region operation.
• Log query incomplete check and automatically query cross pagination.
• Multiple confidential storage types, from file, commandline to env variables.
• Support command line based or file based inputs, complete formation validations.
• Support JMES filter to do further process on results, e.g. select specific fields from json.
• Cross platforms support (Windows, Linux and Mac), Python based and friendly to Py2 and Py3 even Pypy. Support Pip installation.

Installation

Operation System

The CLI supports below operation system:

• Windows
• Mac
• Linux

Supported Version

Python 2.6, 2.7, 3.3, 3.4, 3.5, 3.6, PyPy, PyPy3

Installation Method

Run below command to install the CLI:

> pip install -U aliyun-log-cli

Note

On mac it’s recommended to use pip3 to install the CLI.

> brew install python3
> pip3 install -U aliyun-log-cli

if you encounter errors like OSError: [Errno 1] Operation not permitted, try to use option --user to install:

> pip3 install -U aliyun-log-cli --user

Alicloud ECS which may have limited internet access

You could try the mirrors of local network provider, for Alicloud ECS, you can try below noe:

pip/pip3 install -U aliyun-log-cli --index http://mirrors.aliyun.com/pypi/simple/ --trusted-host mirrors.aliyun.com

Offline Installation

Since 0.1.12, we provide offline package for mac x64 and linux x64 platform.

Follow below ways to install it. 1. download the package from release page 2. unzip it to a local folder, like cli_packages, you can see some whl files inside it. 3. if you don’t have pip, install pip first:

python pip-10.0.1-py2.py3-none-any.whl/pip install --no-index cli_packages/pip-10.0.1-py2.py3-none-any.whl

4. install the CLI:

pip install aliyun-log-cli --no-index --find-links=cli_packages

5. verify it

   > aliyunlog --version
   

FAQ of Installation

1. Encoutering errr TLSV1_ALERT_PROTOCOL_VERSION when installing CLI：

> pip install aliyun-log-cli

Collecting aliyun-log-cli
  Could not fetch URL https://pypi.python.org/simple/aliyun-log-cli/: There was a problem confirming the ssl certificate: [SSL: TLSV1_ALERT_PROTOCOL_VERSION] tlsv1 alert protocol version (_ssl.c:590) - skipping
  Could not find a version that satisfies the requirement aliyun-log-cli (from versions: )
No matching distribution found for aliyun-log-cli

Solution： Please upgrade pip and retry:

pip install pip -U

2. On Linux/Mac, cannot find command aliyunlog?

it’s caused by the missing of shell of aliyunlog, you could make one by yourself.

2.1. find python path:

for linux or mac:

which python

2.2. create a shell script named aliyunlog with below content and allow to execute it. And put it into path folder:

#!<python path here with ! ahead>
import re
import sys
from pkg_resources import load_entry_point

if __name__ == '__main__':
    sys.argv[0] = re.sub(r'(-script\.pyw?|\.exe)?$', '', sys.argv[0])
    sys.exit(
        load_entry_point('aliyun-log-cli', 'console_scripts', 'aliyunlog')()
    )

for linux or mac, it could be put under /usr/bin/.

2.3. verify it

> aliyunlog --version

3. Fail to install module regex?

Refer to below link to install python-devel via yun, apt-get or manually. https://rpmfind.net/linux/rpm2html/search.php?query=python-devel

Full Usage list

Run below command to get the full usage list:

> aliyunlog --help

Note: aliyun command is deprecated to prevent conflict with Universal Ali-cloud CLI.

it will show the full usage.

Note aliyunlog is recommended in case the aliyun conflict with others.

Configure CLI

Refer to Configure CLI.

Input and Output

Inputs

1. Normally case:

> aliyunlog log get_logs --request="{\"topic\": \"\", \"logstore\": \"logstore1\", \"project\": \"dlq-test-cli-123\", \"toTime\": \"2018-01-01 10:10:10\", \"offset\": \"0\", \"query\": \"*\", \"line\": \"10\", \"fromTime\": \"2018-01-01 08:08:08\", \"reverse\":\"false\"}"

2. Input via file: You could store the content of one parameter into a file and pass it via the command line with prefix file://:

> aliyunlog log get_logs --request="file://./get_logs.json"

the content in file get_logs.json as below. Note: the \ is unnecessary to escape the “.

{
  "topic": "",
  "logstore": "logstore1",
  "project": "project1",
  "toTime": "2018-01-01 11:11:11",
  "offset": "0",
  "query": "*",
  "line": "10",
  "fromTime": "2018-01-01 10:10:10",
  "reverse": "true"
}

Parameter Validation

• Mandatory check: if one mandatory parameter is missed, it will report error with usage info.
• Format of parameter’s value will be validated. e.g. int, bool, string list, special data structure.
• for boolean, it support:
• true (case insensitive), T, 1
• false (case insensitive), F, 0
• String list support as [“s1”, “s2”]

Output

1. For operations like Create, Update and Delete, there’s no output except the exit code is 0 which means success.
2. For operations like Get and List, it will output in json format.
3. For errors, it will report in json format as below:

{
  "errorCode":"...",
  "errorMessage":"..."
}

Filter output

It’s supported to filter output via JMES:

Examples:

> aliyunlog log get_logs ...

which outputs:

[ {"__source__": "ip1", "key": "log1"}, {"__source__": "ip2", "key": "log2"} ]

You could use below --jmes-filter to break log into each line:

> aliyunlog log get_logs ... --jmes-filter="join('\n', map(&to_string(@), @))"

output:

{"__source__": "ip1", "key": "log1"}
{"__source__": "ip2", "key": "log2"}

Further Process

You could use >> to store the output to a file. or you may want to process the output using your own cmd. For example, there’s another way to if you may want to break the logs into each line. you could append thd command with a | on linux/unix:

| python2 -c "from __future__ import print_function;import json;map(lambda x: print(json.dumps(x).encode('utf8')), json.loads(raw_input()));"
or
| python3 -c "import json;list(map(lambda x: print(json.dumps(x)), json.loads(input())));"

e.g.

aliyunlog log get_log .... | | python2 -c "from __future__ import print_function;import json;map(lambda x: print(json.dumps(x).encode('utf8')), json.loads(raw_input()));" >> data.txt

Command Reference

Command Specification

1. aliyunlog log <subcommand> [parameters | global options]
2. aliyunlog configure <access_id> <access-key> <endpoint>
3. aliyunlog [--help | --version]

Alias

There’s also an alias aliyunlog for the CLI in case the command aliyun conflicts with others.

1. aliyunlog log <subcommand> [parameters | global options]
2. aliyunlog configure <access_id> <access-key> <endpoint>
3. aliyunlog [--help | --version]

Subcommand and parameters

Actually, the CLI leverage aliyun-log-python-sdk, which maps the command into the methods of aliyun.log.LogClient. The parameters of command line is mapped to the parameters of methods. For the detail spec of parameters, please refer to the Mapped Python SDK API Spec

Examples:

def create_logstore(self, project_name, logstore_name, ttl=2, shard_count=30):

Mapped to CLI:

> aliyunlog log create_logstore
  --project_name=<value>
  --logstore_name=<value>
  [--ttl=<value>]
  [--shard_count=<value>]

Global options

All the commands support below optional global options:

[--access-id=<value>]
[--access-key=<value>]
[--region-endpoint=<value>]
[--client-name=<value>]
[--jmes-filter=<value>]

Command categories

1. Project management
2. Logstore management
3. Shard management
4. Machine group management
5. Logtail config management
6. Machine group and Logtail Config Mapping
7. Index management
8. Cursor management
9. Logs write and consume
10. Shipper management
11. Consumer group management
12. Elasticsearch data migration

1. Project management

• list_project
• create_project
• get_project
• delete_project
• copy_project
• copy all configurations including logstore, logtail, and index config from project to another project which could be in different region.

> aliyunlog log copy_project --from_project="p1" --to_project="p1" --to_client="account2"

• Note: to_client is another account configured via aliyunlog configure, it’s OK to pass main or not to copy inside the same region.
• Refer to Copy project settings cross regions to learn more.

2. Logstore management

• create_logstore
• delete_logstore
• get_logstore
• update_logstore
• list_logstore

3. Shard management

• list_shards
• split_shard
• merge_shard

4. Machine group management

• create_machine_group
• Format of partial parameter:

{
 "machine_list": [
   "machine1",
   "machine2"
 ],
 "machine_type": "userdefined",
 "group_name": "group_name2",
 "group_type": "Armory",
 "group_attribute": {
   "externalName": "ex name",
   "groupTopic": "topic x"
 }
}

• delete_machine_group
• update_machine_group
• get_machine_group
• list_machine_group
• list_machines

5. Logtail config management

• create_logtail_config
• 参考Create Logtail Configuration了解如何创建各种格式的Logtail配置.
• update_logtail_config
• delete_logtail_config
• get_logtail_config
• list_logtail_config

6. Machine group and Logtail Config Mapping

• apply_config_to_machine_group
• remove_config_to_machine_group
• get_machine_group_applied_configs
• get_config_applied_machine_groups

7. Index management

• create_index
• Format of partial parameter:

{
 "keys": {
   "f1": {
     "caseSensitive": false,
     "token": [
       ",",
       " ",
       "\"",
       "\"",
       ";",
       "=",
       "(",
       ")",
       "[",
       "]",
       "{",
       "}",
       "?",
       "@",
       "&",
       "<",
       ">",
       "/",
       ":",
       "\n",
       "\t"
     ],
     "type": "text",
     "doc_value": true
   },
   "f2": {
     "doc_value": true,
     "type": "long"
   }
 },
 "storage": "pg",
 "ttl": 2,
 "index_mode": "v2",
 "line": {
   "caseSensitive": false,
   "token": [
     ",",
     " ",
     "\"",
     "\"",
     ";",
     "=",
     "(",
     ")",
     "[",
     "]",
     "{",
     "}",
     "?",
     "@",
     "&",
     "<",
     ">",
     "/",
     ":",
     "\n",
     "\t"
   ]
 }
}

• update_index
• delete_index
• get_index_config
• list_topics

8. Cursor management

• get_cursor
• get_cursor_time
• get_previous_cursor_time
• get_begin_cursor
• get_end_cursor

9. Logs write and consume

• put_logs
• Format of parameter:

{
"project": "dlq-test-cli-35144",
"logstore": "logstore1",
"topic": "topic1",
"source": "source1",
"logtags": [
  [
    "tag1",
    "v1"
  ],
  [
    "tag2",
    "v2"
  ]
],
"hashKey": "1231231234",
"logitems": [
  {
    "timestamp": 1510579341,
    "contents": [
      [
        "key1",
        "v1"
      ],
      [
        "key2",
        "v2"
      ]
    ]
  },
  {
    "timestamp": 1510579341,
    "contents": [
      [
        "key3",
        "v3"
      ],
      [
        "key4",
        "v4"
      ]
    ]
  }
]
}

• get_logs
• Format of parameter:

{
"topic": "",
"logstore": "logstore1",
"project": "dlq-test-cli-35144",
"toTime": "2018-01-01 11:11:11",
"offset": "0",
"query": "*",
"line": "10",
"fromTime": "2018-01-01 10:10:10",
"reverse": "true"
}

• It will fetch all data when line is passed as -1. But if have large volume of data exceeding 1GB, better to use get_log_all
• get_log_all
• this API is similar as get_logs, but it will fetch data iteratively and output them by chunk. It’s used for large volume of data fetching.
• get_histograms
• pull_logs
• pull_log
• this API is similar as pull_logs, but it allow readable parameter and allow to fetch data iteratively and output them by chunk. It’s used for large volume of data fetching.
• pull_log_dump
• this API will dump data from all shards to local files concurrently.

10. Shipper management

• create_shipper
• Format of partial parameter:

{
"oss_bucket": "dlq-oss-test1",
"oss_prefix": "sls",
"oss_role_arn": "acs:ram::1234:role/aliyunlogdefaultrole",
"buffer_interval": 300,
"buffer_mb": 128,
"compress_type": "snappy"
}

• update_shipper
• delete_shipper
• get_shipper_config
• list_shipper
• get_shipper_tasks
• retry_shipper_tasks

11. Consumer group management

• create_consumer_group
• update_consumer_group
• delete_consumer_group
• list_consumer_group
• update_check_point
• get_check_point

12. Elasticsearch data migration

• migrate
• please refer to Elasticsearch data migration to know how to migrate data from elasticsearch to aliyun log service.

Best Practice

• Configure CLI
• Create Logtail Config
• Duplicate project settings cross region
• Pull Logs
• Get Logs
• Elasticsearch data migration
• Flexible Datatime format
• Copy Data cross region & Reindex data

Troubleshooting

By default, CLI store erros or warnings at ~/aliyunlogcli.log, it’s also configurable via file ~/.aliyunlogcli, section __loggging__ to adjust the logging level and location:

[__logging__]
filename=  # default: ~/aliyunlogcli.log, Rotated when hit filebytes
filebytes=   # Deafult: 5242880 (5MB), file size of each log before rotation, Unit: Bytes
backupcount= # Default: 5, file backup file
#filemode=  # deprecated
format=    # default: %(asctime)s %(levelname)s %(filename)s:%(lineno)d %(funcName)s %(message)s
datefmt=   # default: "%Y-%m-%d %H:%M:%S", could be strftime() compitable date/time formatting string
level=     # default: warn, could be: info, error, fatal, critical, debug

Other resources

1. Alicloud Log Service homepage：https://www.alibabacloud.com/product/log-service
2. Alicloud Log Service doc：https://www.alibabacloud.com/help/product/28958.htm
3. Alicloud Log Python SDK doc: http://aliyun-log-python-sdk.readthedocs.io/
4. for any issues, please submit support tickets

用户手册

README in English

内容

• 简介
  • 基本介绍
  • 主要功能
• 安装
  • 操作系统
  • 支持版本
  • 安装方式
  • 离线安装
  • 安装常见问题
  • 完整参数列表
• 配置说明
• 输入输出
  • 输入
  • 参数校验
  • 输出
  • 输出过滤
  • 进一步处理
• 命令参考
  • 命令规范
  • 命令别名
  • 子命令与参数
  • 全局选项
  • 命令类别
    • 项目组管理
    • 日志库管理
    • 分区管理
    • 机器组管理
    • Logtail配置管理
    • 机器组与Logtail配置关联
    • 索引管理
    • 游标操作
    • 日志读写与消费
    • 投递管理
    • 消费组管理
    • Elasticsearch数据迁移
    • 其他配置管理
• 最佳实践
• 错误诊断
• 其他资源

简介

日志服务(SLS)提供了Web和SDK的方式进行日志采集、管理、查询和分析。为了满足越来越多的的自动化日志配置查询分析的需求，现重磅发布SLS的命令行工具(Command Line Interface - CLI)。

基本介绍

日志服务命令行工具CLI支持几乎所有操作，日志查询支持完整性检查与自动分页、支持多账户与跨域复制。

主要功能

• 支持大部分的日志服务REST接口。
• 支持多账号，方便测试与跨域操作。
• 日志查询完整性查询与自动分页。
• 支持多种方式存储和配置秘钥，适用于各种情况。
• 支持通过命令行或者文件的形式输入复杂参数，并校验参数内容格式。
• 支持JMES过滤器对结果进行进一步处理，方便选择特定信息。
• 跨平台（Windows/Linux和Mac）支持，基于Python（Py2.6+和Py3.3+平台友好），支持Pip安装。

安装

操作系统

日志服务CLI支持以下操作系统：

• Windows
• Mac OS
• Linux

支持版本

Python 2.6、2.7、3.3、3.4、3.5、3.6、PyPy和PyPy3。

安装方式

执行以下命令安装日志服务CLI。

> pip install -U aliyun-log-cli

注意

Mac上推荐使用pip3安装CLI，首选需要安装Python3：

> brew install python3
> pip3 install -U aliyun-log-cli

Mac上如果你安装时遇到了权限相关的错误，如：OSError: [Errno 1] Operation not permitted，尝试使用如下命令安装：

> pip3 install -U aliyun-log-cli --user

外网访问受限的阿里云ECS

因为安全策略等原因，某些云服务器可能无法访问Pypi服务器，可以尝试使用本地Pypi镜像，例如阿里云服务器可以尝试:

pip/pip3 install -U aliyun-log-cli --index http://mirrors.aliyun.com/pypi/simple/ --trusted-host mirrors.aliyun.com

Note：也可以使用清华的索引-i https://pypi.tuna.tsinghua.edu.cn/simple

离线安装

从 0.1.12开始, 我们为Linux x64和MacOS x64提供离线的安装包.

如下步骤安装:

1. 从release页面下载对应离线安装包.
2. 解压到本地目录, 例如cli_packages, 可以看到许多whl文件在里面.
3. 如果还没有pip, 可以如下安装:

python pip-10.0.1-py2.py3-none-any.whl/pip install --no-index cli_packages/pip-10.0.1-py2.py3-none-any.whl

4. 安装CLI:

pip install aliyun-log-cli --no-index --find-links=cli_packages

5. 验证:

   > aliyunlog --version
   

常见安装问题

1. 安装时报TLSV1_ALERT_PROTOCOL_VERSION错误

> pip install aliyun-log-cli

Collecting aliyun-log-cli
  Could not fetch URL https://pypi.python.org/simple/aliyun-log-cli/: There was a problem confirming the ssl certificate: [SSL: TLSV1_ALERT_PROTOCOL_VERSION] tlsv1 alert protocol version (_ssl.c:590) - skipping
  Could not find a version that satisfies the requirement aliyun-log-cli (from versions: )
No matching distribution found for aliyun-log-cli

解答： 请先升级pip：

pip install pip -U

2. 找不到命令aliyunlog?

Linux/Mac下, 因为某种原因创建脚本aliyunlog链接时未成功, 可以手工创建一个, 如下:

2.1. 找到Python路径:

在linux或mac, 执行命令可以看到:

which python

2.2. 创建一个叫做aliyunlog文件, 赋予执行权限, 内容如下, 并放到PATH目录下 :

#!<python路径放这里,注意有一个感叹号!>
import re
import sys
from pkg_resources import load_entry_point

if __name__ == '__main__':
    sys.argv[0] = re.sub(r'(-script\.pyw?|\.exe)?$', '', sys.argv[0])
    sys.exit(
        load_entry_point('aliyun-log-cli', 'console_scripts', 'aliyunlog')()
    )

对于linux或mac, 可以考虑放到/usr/bin/目录下

2.3. 验证, 执行如试下命令看一下是否成功. ```shell > aliyunlog –version

3. Regex模块安装失败? 如果存在安装Regex失败的错误, 可以参考使用yun/apt-get或者手动安装一下python-devel https://rpmfind.net/linux/rpm2html/search.php?query=python-devel

完整参数列表

执行以下命令查看日志服务CLI参数列表。

> aliyunlog --help

注意: 命令aliyun已经废弃(为了避免与通用阿里云CLI冲突).

会显示完整命令。

Note aliyunlog和aliyun都可以，推荐使用aliyunlog以防冲突。

配置说明

参考配置CLI

输入输出

输入

1. 一般输入

> aliyunlog log get_logs --request="{\"topic\": \"\", \"logstore\": \"logstore1\", \"project\": \"dlq-test-cli-123\", \"toTime\": \"2018-01-01 11:11:11\", \"offset\": \"0\", \"query\": \"*\", \"line\": \"10\", \"fromTime\": \"2018-01-01 10:10:10\", \"reverse\":\"false\"}"

2. 文件输入

也可以将上面参数放到一个文件里面, 简化命令行, 需要义file://开头+文件路径即可:

> aliyunlog log get_logs --request="file://./get_logs.json"

文件get_logs.json内容如下, 注意: 文件中不需要反斜杠\来转义.

{
  "topic": "",
  "logstore": "logstore1",
  "project": "project1",
  "toTime": "2018-01-01 11:11:11",
  "offset": "0",
  "query": "*",
  "line": "10",
  "fromTime": "2018-01-01 10:10:10",
  "reverse": "true"
}

参数校验

• 必填的参数没有填写时会报错, 输出参数列表
• 参数格式本身会进行校验, 例如int, bool, string list, 特定数据结构等
• bool支持的形式有:
• true (大小写不敏感), T, 1
• false (大小写不敏感), F, 0
• 字符串列表支持的形式为[“s1”, “s2”]

输出

1. 对于Create, Update, Delete操作, 一般脚本无输出, exit code=0表示成功.
2. 对于Get/List操作, 以json格式输出内容
3. 错误情况下, 以如下格式返回错误:

{
  "errorCode":"...",
  "errorMessage":"..."
}

输出过滤

支持通过JMES过滤输出的结果.

例如:

> aliyunlog log get_logs ...

以上命令的输出是:

[ {"__source__": "ip1", "key": "log1"}, {"__source__": "ip2", "key": "log2"} ]

通过以下命令将日志分隔为每一行:

> aliyunlog log get_logs ... --jmes-filter="join('\n', map(&to_string(@), @))"

输出:

{"__source__": "ip1", "key": "log1"}
{"__source__": "ip2", "key": "log2"}

进一步处理

你可以使用>>来讲输出存储到一个文件. 某些时候, 你需要使用其他命令进行处理, 例如, 这里介绍另一个把json格式的日志分行打印的方法. 在Linux/Unix下, 你可以在命令后通过添加一个|来进一步处理.

| python2 -c "from __future__ import print_function;import json;map(lambda x: print(json.dumps(x).encode('utf8')), json.loads(raw_input()));"
or
| python3 -c "import json;list(map(lambda x: print(json.dumps(x)), json.loads(input())));"

例如:

aliyunlog log get_log .... |  python2 -c "from __future__ import print_function;import json;map(lambda x: print(json.dumps(x).encode('utf8')), json.loads(raw_input()));" >> data.txt

命令参考

命令规范

1. aliyunlog log <subcommand> [parameters | global options]
2. aliyunlog configure <access_id> <access-key> <endpoint> [<client-name>]
3. aliyunlog [--help | --version]

命令别名

日志服务的CLI也有一个别名aliyunlog, 如果aliyun这个命令冲突了, 可以尝试使用aliyunlog:

1. aliyunlog log <subcommand> [parameters | global options]
2. aliyunlog configure <access_id> <access-key> <endpoint>
3. aliyunlog [--help | --version]

子命令与参数

日志服务命令行工具背后依赖于日志服务的Python SDK, 相关子命令对应于aliyun.log.LogClient的方法, 参数和可选参数也一一对应. 具体支持的API参数, 请参考映射的Python SDK API

例子:

def create_logstore(self, project_name, logstore_name, ttl=2, shard_count=30):

对应命令行:

> aliyunlog log create_logstore
  --project_name=<value>
  --logstore_name=<value>
  [--ttl=<value>]
  [--shard_count=<value>]

全局选项

所有命令都支持如下的全局选项:

[--access-id=<value>]
[--access-key=<value>]
[--region-endpoint=<value>]
[--client-name=<value>]
[--jmes-filter=<value>]

命令类别

1. 项目组管理
2. 日志库管理
3. 分区管理
4. 机器组管理
5. Logtail配置管理
6. 机器组与Logtail配置关联
7. 索引管理
8. 游标操作
9. 日志读写与消费
10. 投递管理
11. 消费组管理
12. Elasticsearch 数据迁移

1. 项目组管理

• list_project
• create_project
• get_project
• delete_project
• copy_project
• 复制所有源project的logstore, logtail, machine group和index配置等到目标project中.

> aliyunlog log copy_project --from_project="p1" --to_project="p1" --to_client="account2"

• 注意: to_client是通过aliyunlog configure配置的其他账户, 也可以不传或传main同域复制.
• 参考跨域复制项目组配置了解如何跨域复制项目组配置.

2. 日志库管理

• create_logstore
• delete_logstore
• get_logstore
• update_logstore
• list_logstore

3. 分区管理

• list_shards
• split_shard
• merge_shard

4. 机器组管理

• create_machine_group
• 部分参数格式:

{
 "machine_list": [
   "machine1",
   "machine2"
 ],
 "machine_type": "userdefined",
 "group_name": "group_name2",
 "group_type": "Armory",
 "group_attribute": {
   "externalName": "ex name",
   "groupTopic": "topic x"
 }
}

• delete_machine_group
• update_machine_group
• get_machine_group
• list_machine_group
• list_machines

5. Logtail配置管理

• create_logtail_config
• 参考创建Logtail配置了解如何创建各种格式的Logtail配置.
• update_logtail_config
• delete_logtail_config
• get_logtail_config
• list_logtail_config

6. 机器组与Logtail配置关联

• apply_config_to_machine_group
• remove_config_to_machine_group
• get_machine_group_applied_configs
• get_config_applied_machine_groups

7. 索引管理

• create_index
• 部分参数格式:

{
 "keys": {
   "f1": {
     "caseSensitive": false,
     "token": [
       ",",
       " ",
       "\"",
       "\"",
       ";",
       "=",
       "(",
       ")",
       "[",
       "]",
       "{",
       "}",
       "?",
       "@",
       "&",
       "<",
       ">",
       "/",
       ":",
       "\n",
       "\t"
     ],
     "type": "text",
     "doc_value": true
   },
   "f2": {
     "doc_value": true,
     "type": "long"
   }
 },
 "storage": "pg",
 "ttl": 2,
 "index_mode": "v2",
 "line": {
   "caseSensitive": false,
   "token": [
     ",",
     " ",
     "\"",
     "\"",
     ";",
     "=",
     "(",
     ")",
     "[",
     "]",
     "{",
     "}",
     "?",
     "@",
     "&",
     "<",
     ">",
     "/",
     ":",
     "\n",
     "\t"
   ]
 }
}

• update_index
• delete_index
• get_index_config
• list_topics

8. 游标操作

• get_cursor
• get_cursor_time
• get_previous_cursor_time
• get_begin_cursor
• get_end_cursor

9. 日志读写与消费

• put_logs
• 参数格式:

{
"project": "dlq-test-cli-35144",
"logstore": "logstore1",
"topic": "topic1",
"source": "source1",
"logtags": [
  [
    "tag1",
    "v1"
  ],
  [
    "tag2",
    "v2"
  ]
],
"hashKey": "1231231234",
"logitems": [
  {
    "timestamp": 1510579341,
    "contents": [
      [
        "key1",
        "v1"
      ],
      [
        "key2",
        "v2"
      ]
    ]
  },
  {
    "timestamp": 1510579341,
    "contents": [
      [
        "key3",
        "v3"
      ],
      [
        "key4",
        "v4"
      ]
    ]
  }
]
}

• get_logs
• 参数格式:

{
"topic": "",
"logstore": "logstore1",
"project": "dlq-test-cli-35144",
"toTime": "2018-01-01 11:11:11",
"offset": "0",
"query": "*",
"line": "10",
"fromTime": "2018-01-01 10:10:10",
"reverse": "true"
}

• 但参数line传入-1时, 就回获取所有. 但是当数据量很大, 超过1GB时, 最好使用get_log_all
• get_log_all
• 和get_logs一样, 但是迭代式获取数据并输出, 适合大容量的数据获取.
• get_histograms
• pull_logs
• pull_log
• 和pull_logs类似, 但是迭代式获取数据并输出, 适合大容量的数据获取.
• pull_log_dump
• 并发下载大量日志直接到本地文件，并一行一个日志。

10. 投递管理

• create_shipper
• 部分参数格式:

{
"oss_bucket": "dlq-oss-test1",
"oss_prefix": "sls",
"oss_role_arn": "acs:ram::1234:role/aliyunlogdefaultrole",
"buffer_interval": 300,
"buffer_mb": 128,
"compress_type": "snappy"
}

• update_shipper
• delete_shipper
• get_shipper_config
• list_shipper
• get_shipper_tasks
• retry_shipper_tasks

11. 消费组管理

• create_consumer_group
• update_consumer_group
• delete_consumer_group
• list_consumer_group
• update_check_point
• get_check_point

12. Elasticsearch 数据迁移

• es_migrate
• 参考Elasticsearch 数据迁移了解如何将 Elasticsearch 中的数据导入日志服务。

13. 其他配置管理

• 报警管理
• 快速查询管理
• 报表管理
• 外部存储管理

参考命令行帮助

最佳实践

• 配置CLI
• 创建Logtail配置
• 跨域复制项目配置
• 拉取日志
• 查询日志
• Elasticsearch 数据迁移
• 支持的灵活时间格式
• 高速跨域日志复制、历史数据重新索引与数仓投递

错误诊断

CLI默认把执行过程中的警告和错误存储在~/aliyunlogcli.log中, 也可以配置~/.aliyunlogcli中__loggging__来调整错误存储位置和格式:

[__logging__]
filename=  # 默认是: ~/aliyunlogcli.log, 进行轮转
filebytes=   # 默认是5242880 (5MB), 单个文件大小, 单位Byte
backupcount= # 默认是5个, 轮转文件的个数
#filemode=  # 废弃
format=    # 默认是: %(asctime)s %(threadName)s:%(levelname)s %(filename)s:%(lineno)d %(funcName)s %(message)s
datefmt=   # 默认是: "%Y-%m-%d %H:%M:%S", 可以是strftime()兼容的日期/时间格式化字符串
level=     # 默认是: warn, 可以是info, error, fatal, critical, debug

其他资源

1. 日志服务产品介绍：http://www.aliyun.com/product/sls/
2. 日志服务产品文档：https://help.aliyun.com/product/28958.html
3. 日志服务Python SDK文档: http://aliyun-log-python-sdk.readthedocs.io/
4. 其他问题请提工单

Best Practices

Configure CLI

Introduction

There’re ceratin options need to configure to make it more convenient to use CLI.

CLI options

CLI options include:

• Access Key and Endpoint
  • About the definition of access key and endpoint，Refer to here
• Multiple account
  • Configure multiple accounts for operation in specific region, cross region or testing purpose.
• Output format
  • format the json output.

Configure AK and Endpoint

There’re three ways to configure the access key and endpoint and they’re prioritized as below:

• Parameters

> aliyunlog log create_project ..... --access-id=<value> --access-key=<value> --region-endpoint=<value>

Note: Any sub command support such way to overwrite the AK setings in later ways (env or config file) for the specific operations.

• Environment Variables
• ALIYUN_LOG_CLI_ACCESSID
• ALIYUN_LOG_CLI_ACCESSKEY
• ALIYUN_LOG_CLI_ENDPOINT
• Local configuration file

You could store them at ~/.aliyunlogcli, the default section name is main

[main]
access-id=
access-key=
region-endpoint=

Enable Https Connection

When configuring endpoint with prefix https://, the connection between CLI and Log service will be secured. Or else, it will use http by default.

Modify the configuration file

Use the command “configure” to modify the configuration file:

> aliyunlog configure access_id access_key cn-beijing.log.aliyuncs.com

Multiple Account

1. Store multiple accounts for some use cases (e.g. test, multiple region operations)

> aliyunlog configure access_id1 access_key1 cn-beijing.log.aliyuncs.com
> aliyunlog configure access_id2 access_key2 cn-hangzhou.log.aliyuncs.com test

AK is stored as:

[main]
access-id=access_id1
access-key=access_key1
region-endpoint=cn-beijing.log.aliyuncs.com

[test]
access-id=access_id2
access-key=access_key2
region-endpoint=cn-hangzhou.log.aliyuncs.com

Refer to here to know a simpler way to configure multiple accounts in config file.

2. Use specific account

Any subcommand could use global opton --client-name=<value> to use specific configured account. e.g:

> aliyunlog log create_project ..... --client-name=test

It will use test to create the project.

3. Other Case

In some case, we need to operate cross regions, e.g.

> aliyunlog log copy_project --from_project="p1" --to_project="p1" --to_client=test

It will use account main to copy project p1 in its region to another region under account test

Default account

As mentions previously, the name of default account is main, in specific case, it’s necessary to swith to another account (in different region). This could be archvied by using option --default-client:

aliyunlog configure --default-client=beijing

This command will switch default account to the one beijing.

Format output

JSON Format

Output is in json format normally, which is printed in one line in console. In some cases, it’s useful to show them in pretty format. You could use option --format-output=json in each command for this:

aliyunlog log get_log....  --format-output=json

But if you want to make it a default behavior, configure it directly:

aliyunlog log configure --format-output=json

Not escape Non-ANSI

By default, for non-ANSI characters, it will escape it. If you want to view the raw string, you could add a no_escape to format of output:

aliyunlog log get_log....  --format-output=json,no_escape

It could be no_escape or combine with other value in format-output with ,

And if you want to make it a default behavior, configure it directly:

aliyunlog log configure --format-output=json,no_escape

Get logs

Introduction

Get Log (GetLog) means searching out logs on certain conditions among specific time range basing on indexed data.

It has below characters:

1. Just for index data.
2. Against index and cross partition. Support get logs cross multiple logstores.
3. Support search against topic, fulltext or fields etc.
4. The time range is checking the logs’ time rather than the log rececieved time by server.

Refer to Query logs to learn how to turn on index and more syntax of searching.

Get log

Using CLI, it’s very easy to query logs basing on prepared search statements. Below is an example, it all slow access logs on a specific server among a specific time range.

aliyunlog log get_log_all --project="p1" --logstore="l1" --query="host:test.com and response_time>5000" --from_time="2018-01-24 16:00:00 CST" --to_time="2018-01-24 17:00:00  CST"

It get all logs that his the conditions (host:test.com and response_time>5000) from time range (from 2018-01-24 16:00:00 CST to 2018-01-24 17:00:00 CST) like:

[{"k1":"a1", "k2":"a2"}, {"k1":"b1", "k2":"b2"}, ... ]

Note: - We could also use another sub-command get_log(s) passing parameter size=-1 to get all logs hitting the condition. But it’s recommended to use the sub-command get_log_all when the data volume is large (e.g. total logs count exceeds 1 million).

Transform the format and store to file

The example above shows the format of get log is a big JSON formatted data, if we want to store the logs per row per log, we need to pass parameter jmes-filter to process and use redirection >> to output to disk.

Examples:

aliyunlog log get_log_all --project="p1" --logstore="l1" --query="host:test.com and response_time>5000" --from_time="2018-01-24 16:00:00 CST" --to_time="2018-01-24 17:00:00  CST" --jmes-filter="join('\n', map(&to_string(@), @))" >> ~/Desktop/test.data

It will store the results to file test.data in below format:

{"k1":"a1", "k2":"a2"}
{"k1":"b1", "k2":"b2"}
...

Time format

The time format passed to the CLI is recommend as %Y-%m-%d %H:%M:%S %Z, like 2018-01-24 17:00:00 CST. But the CLI also support popular format like Jan 01 2018 10:10:10 CST

Note： The time-zone could not only be CST but also any other valid ones. Try to use more accurate time zone format like UTC-8 or UTC+8 when you could not find the wanted time.

Actually, it’s also supported to use relative format like 2 day ago, now, yesterday 0:0, -1week etc.

Time range

When passing time range parameters to get_log(s) or get_log_all, please note:

1. The time here points to log time parsed from raw logs when logs are uploaded (If not being specified, time received by server is used).
2. The time range is closed for both left and right as [], it means it will get logs at 16:00:00 and 17:00:00 in above example.

Query cross logstores

To get logs cross logstores, use sub-command get_project_logs, example:

aliyunlog log get_project_logs --request="{\"project\":\"p1\", \"query\":\"select count(1) from logstore1, logstore2, logstore3 where __date__ >'2017-11-10 00:00:00' and __date__ < '2017-11-13 00:00:00'\"}"

For more detail, please refer to document Query cross logstore.

最佳实践

配置CLI

介绍

和大多数CLI一样，CLI安装好后，需要适当配置一下才可以高效方便的使用。

有哪些CLI的配置项

CLI的配置项目包含如下内容:

• 服务入口和访问秘钥
  • 关于服务入口和访问秘钥的定义，可以参考这里
• 多账户与默认账户
  • 配置多个账户，以便访问特定区域，用于测试或者跨域操作等。
• 输出格式
  • 修改输出json的格式化方式，默认是在一行打印。
  • 修改输出非英文转义方式，默认转义.

配置秘钥与服务入口

日志服务CLI支持通过以下三种方式配置访问秘钥与入口，如果同时配置多种方式, 优先顺序是: 参数, 环境变量, 最后是本地配置文件.

• 参数

> aliyunlog log create_project ..... --access-id=<value> --access-key=<value> --region-endpoint=<value>

注意: 任意log子命令都支持以上方式定义特定的AK与Endpoint(覆盖后面的方式)

• 环境变量
• ALIYUN_LOG_CLI_ACCESSID
• ALIYUN_LOG_CLI_ACCESSKEY
• ALIYUN_LOG_CLI_ENDPOINT
• 本地配置文件

将存储AK与Endpoint在~/.aliyunlogcli, 默认使用的块名是main

[main]
access-id=
access-key=
region-endpoint=

使用HTTPS连接

当给服务入口地址加上https://前缀时, CLI会使用https连接, 否则默认使用http.

修改配置文件

Configure命令可以修改配置文件内容.

> aliyunlog configure access_id access_key cn-beijing.log.aliyuncs.com

多账户

存储于多个账户

CLI支持以便在特定情况下使用(例如不同域操作、测试等):

> aliyunlog configure access_id1 access_key1 cn-beijing.log.aliyuncs.com
> aliyunlog configure access_id2 access_key2 cn-hangzhou.log.aliyuncs.com test

AK将存储为:

[main]
access-id=access_id1
access-key=access_key1
region-endpoint=cn-beijing.log.aliyuncs.com

[test]
access-id=access_id2
access-key=access_key2
region-endpoint=cn-hangzhou.log.aliyuncs.com

参考这里来了解一种简化的多账户配置方式。

使用特定账户

任意命令都可以通过选项--client-name=<value>来使用特定配置的账户, 例如:

> aliyunlog log create_project ..... --client-name=test

将使用test的AK来进行操作.

其他情况

某些情况下也需要跨账户操作, 例如:

> aliyunlog log copy_project --from_project="p1" --to_project="p1" --to_client=test

将main账户下对应的项目p1复制到账户test下的p1

默认账户

如前面所属，默认账户是main，在多用户情况下，切换默认账户也是很有用。通过选项--default-client可以修改。

aliyunlog configure --default-client=beijing

这样其他命令默认会使用beijing账户下的访问密码和服务入口地址。

输出格式

JSON格式

输出格式一般是以json形式输出，并且是打印在一行上面，某些情况下需要格式化输出方便查看，可以在特定命令上配置--format-output=json，这样CLI会将输出进行格式化。

aliyunlog log get_log....  --format-output=json

如果期望所有输出都是这样，可以修改配置项来完成：

aliyunlog log configure --format-output=json

不转义非ANSI字符

对于非英文字符, 默认是转义的形式. 如果期望可以看到原始字符, 可以添加no_escape到format-ouput选项中:

aliyunlog log get_log....  --format-output=json,no_escape

与其他选项用逗号分隔即可. 如果期望所有输出都是这样，可以修改配置项来完成：

aliyunlog log configure --format-output=json,no_escape

使用CLI了解基础资源使用状况

背景

我们知道，日志服务的每个Project有一定的资源限制，你是否时常需要知道一些重要资源的使用状况？例如： 1. 一个Project的总shard数量是否已经超过了200个限制？ 2. Logtail配置、机器组、报表数等是否快要满了？

现在使用CLI一行命令即可获得相关信息，以便及时提工单申请更大限额。

介绍

获取资源使用状况(get_resource_usage)检查传入的Project的所有基础资源，并以Json的形式列出，覆盖如下基础资源：

前提

这里假设已经完成了CLI的安装和配置.

获取基础资源使用状况

一行命令即可获取：

例如：

aliyunlog log get_resource_usage  --project="my project name"  --format-output=json,no_escape

返回的格式是一个json，例如：

{
  "consumer_group": {
    "count": {
      "status": 8
    },
    "logstores": {
      "access-log": {
        "limitation": 10,
        "status": 7,
        "usage": "70.0%"
      },
      "billdata": {
        "limitation": 10,
        "status": 1,
        "usage": "10.0%"
      }
    }
  },
  "dashboard": {
    "count": {
      "limitation": 50,
      "status": 28,
      "usage": "56.0%"
    }
  },
  "logstore": {
    "count": {
      "limitation": 200,
      "status": 16,
      "usage": "8.0%"
    }
  },
  "logtail": {
    "count": {
      "limitation": 100,
      "status": 2,
      "usage": "2.0%"
    }
  },
  "machine_group": {
    "count": {
      "limitation": 100,
      "status": 1,
      "usage": "1.0%"
    }
  },
  "saved_search": {
    "count": {
      "limitation": 100,
      "status": 11,
      "usage": "11.0%"
    }
  },
  "shard": {
    "count": {
      "limitation": 200,
      "status": 30,
      "usage": "15%"
    },
    "logstores": {
      "access-log": {
        "status": 2
      },
      "billdata": {
        "status": 10
      },
      "test": {
        "status": 2
      },
      "testmax": {
        "status": 2
      },
      "tripdata": {
        "status": 10
      },
      "vedio1": {
        "status": 2
      },
      "web-1": {
        "status": 2
      }
    }
  }
}

注意： 1. 返回结构包含字段limitation与usage来表示限制与使用占比，但是这里的是默认的限制，如果通过工单增加过额度，这里并不会更新。 2. 这里传入了参数format-output来格式化json，如果已经配置过整个CLI，可以省略。

单看shard消耗

有时候我们主要关心一个两个核心指标，并可能仅仅想要一个数字，可以通过jmes-filter参数来进行处理，例如这里仅仅查看总的Shard使用情况，可以如下过滤：

aliyunlog log get_resource_usage  --project="my project name" --jmes-filter="shard.count.status"
> 30

这里通过--jmes-filter="shard.count.status直接获取输出结果中的shard的目前总数。

更多关于JMES过滤的信息，请参考这里。

使用CLI配置更多的shard

背景

我们知道，日志服务的每个Project有一定的资源限制，你是否遇到如下情况呢？ 1. 创建Shard时，最多配置10个Shard，但是预期Logstore会有更大数据量写入尤其是查询读取需求？ 2. 手工在Web控制台进行分裂，因为无法把握均匀分布，且点击较多，比较低效？

现在使用CLI一行命令即可配置Logstore到期望的数量了！

介绍

CLI通过命令arrange_shard来自动将目标logstore的shard数量均匀分裂为期望的数量（最多到100个甚至更多）。

前提

这里假设已经完成了CLI的安装和配置.

配置shard数量

例如：

aliyunlog log arrange_shard --project=my_project1 --logstore=my_logstore1 --count=50

这里配置目标logstore的shard数量为50个。

原理与注意事项

命令arrange_shard根据期望shard的数量，以及目前shard分区范围，自动均匀的将目前分裂。

注意： 1. 这个命令也是通过split_shard进行分裂，因此分裂过程中会制造出多个只读的shard。在一个Project最多有200个shard的情况下，如果一个Project只包含一个logstore，这个logstore可以配置为最多100个读写shard。 2. 因为服务器同步的原因，分裂命令完成后，在Web控制台一般需要1分钟左右可以看到最新shard数量。

在配置文件中简单配置多个账户

介绍

这里详细介绍一种通过配置文件，简单的配置多个账户的方式。

背景

在配置CLI里介绍了详细的CLI配置方法。如果要配置多个账户，可以通过CLI的configure命令配置多个账户：

> aliyunlog configure access_id1 access_key1 cn-beijing.log.aliyuncs.com
> aliyunlog configure access_id2 access_key2 cn-hangzhou.log.aliyuncs.com test

AK将存储为:

[main]
access-id=access_id1
access-key=access_key1
region-endpoint=cn-beijing.log.aliyuncs.com

[test]
access-id=access_id2
access-key=access_key2
region-endpoint=cn-hangzhou.log.aliyuncs.com

如果我们有好多个域都需要配置，并且还存在多个权限的账户（如管理员和只读权限的秘钥），那么配置就有些复杂了。

方案

配置文件中，可以在DEFAULT中集中配置多个权限的秘钥，再在其他区中用%(xxxx)s的方式引用它即可，这样可以手动编辑~/.aliyunlogcli来实现：

[DEFAULT]
admin_ak_id = id
admin_ak_key = key
user_ak_id = id
user_ak_key = key

[hz_admin]
access-id = %(admin_ak_id)s
access-key =  %(admin_ak_key)s
region-endpoint = cn-hangzhou.sls.aliyuncs.com

[hz_user]
access-id = %(user_ak_id)s
access-key =  %(user_ak_key)s
region-endpoint = cn-hangzhou.sls.aliyuncs.com

[bj_admin]
access-id = %(admin_ak_id)s
access-key =  %(admin_ak_key)s
region-endpoint = cn-beijing.sls.aliyuncs.com

[bj_user]
access-id = %(user_ak_id)s
access-key =  %(user_ak_key)s
region-endpoint = cn-beijing.sls.aliyuncs.com

使用

可以自由切换默认的账户, 更多参考配置CLI

aliyunlog configure --default-client=bj_admin

创建Logtail配置

介绍

Logtail是日志服务提供的高性能低耗的日志收集客户端, 参考这里了解更多背景. 目前Logtail的配置有多种, 本节介绍如何快速有效的通过CLI创建各种Logtail的配置项.

有哪些Logtail的配置项

Logtail配置项目包含如下内容:

• 基于文本文件的:
• 极简模式
• JSON模式
• 分隔符模式
• 完整正则表达式模式
• syslog
• 自建产品
• NGNIX, 本节通过完整正则表达式来完成
• 飞天日志等

准备

这里假设已经完成了CLI的安装.

配置域账户

首先CLI中配置默认区域和账户, 以便后续操作, 这一步也是一次性的. 如果之前已经做过, 这里可以跳过.

这里配置一个杭州公有云的账户

> aliyunlog configure AKID****123 AKKEY****123 cn-hangzhou.log.aliyuncs.com

参考:

• 关于秘钥的配置, 可以参考配置
• 关于日志服务在各个域的Endpoint地址, 可以参考入口服务

配置

1. 创建极简模式的logtail配置

极简模式是对文件进行单行分隔, 放入到名为content的域中的最简单模式. 适用于简单存储处理的情况.

这里我们在项目组project1中创建一个极简模式的logtail配置:

> aliyunlog log create_logtail_config --project_name="project1" --config_detail="file://simple_1.json"

文件simple_1.json的内容如下:

{
  "configName": "simple_1",
  "inputDetail": {
    "logType": "common_reg_log",
    "logPath": "/user",
    "filePattern": "*.log"
  },
  "inputType": "file",
  "outputDetail": {
    "logstoreName": "logstore1"
  }
}

这里创建了一个属于日志库logstore1的名叫simple_1的logtail的配置. 主要的配置项就是日志的路径logPath以及文件名模式filePattern. 这里扫描/user下包含深度子目录的所有.log文件.

注意: 除了以上项目, 还需要如上所示填入相关信息: 其中inputType是file, inputDetail.logType是common_reg_log.

时间配置

简单模式的日志时间采用的是系统收集的时间, 也就是抓取到这条日志的时间.

2. 创建JSON模式的logtail配置

JSON模式适用于文件本身就是JSON的情况.

这里我们在项目组project1中创建一个JSON模式的logtail配置:

> aliyunlog log create_logtail_config --project_name="project1" --config_detail="file://json_1.json"

文件json_1.json的内容如下:

{
  "configName": "json_1",
  "inputDetail": {
    "logType": "json_log",
    "filePattern": "*.json",
    "logPath": "/json_1"
  },
  "inputType": "file",
  "outputDetail": {
    "logstoreName": "logstore1"
  }
}

这里创建了一个属于日志库logstore1的名叫json_1的logtail的配置. 主要的配置项就是日志的路径logPath以及文件名模式filePattern. 这里扫描/user下包含深度子目录的所有.json文件.

注意:

1. 这里需要按照如上的模板填入相关信息. 其中inputType必须是file, inputDetail.logType是json_log.
2. Logtail会根据json的格式自动提取对应的域信息. 例如一个旅行日志的内容如下:

{
  "from": "nanjing",
  "to": "shanghai",
  "people": "xiaoming",
  "travel_time": "2018-1-1 10:10:10"
}

会自动配置域from, to和people和travel_time.

时间配置

默认情况下, 日志的时间会使用系统时间, 也就是抓取到这条日志的时间. 如果日志本身包含了更加准确的时间, 例如上面旅行日志的例子, 可以在logtail的配置中指定travel_time为日志时间如下:

{
  "inputDetail": {
    "timeFormat": "%Y-%M-%D %h:%m:s",
    "timeKey": "travel_time"
  }
}

这里通过inputDetail.timeKey指定travel_time为时间关键字, 通过inputDetail.timeFormat配置其格式为%Y-%M-%D %h:%m:s, 针对时间格式, 可以进一步参考这里

3. 创建分隔符模式的logtail配置

分隔符模式适用于文件本身基于某个分隔符分隔的文件, 例如CSV, TSV等等.

日志服务支持最多3个字符组合的分隔, 例如一个旅行日志的文件内如是:

2017-1-1 10:10:00&#&nanjing&#&shanghai&#&xiao ming
2017-1-1 20:10:00&#&beijing&#&hangzhou&#&xiao wang

分隔符就是&#&可以分隔出时间, 出发城市, 达到城市, 旅客.

这里我们在项目组project1中创建一个分隔符模式的logtail配置:

> aliyunlog log create_logtail_config --project_name="project1" --config_detail="file://sep_1.json"

文件sep_1.json的内容如下:

{
  "configName": "sep_1",
  "logSample": "2017-1-1 10:10:00&#&nanjing&#&shanghai&#&xiao ming",
  "inputDetail": {
    "logType": "delimiter_log",
    "logPath": "/user",
    "filePattern": "travel.log",
    "separator": "&#&",
    "key": [
      "travel_time",
      "from_city",
      "to_city",
      "people"
    ]
  },
  "inputType": "file",
  "outputDetail": {
    "logstoreName": "logstore1"
  }
}

这里创建了一个属于日志库logstore1的名叫json_1的logtail的配置. 主要的配置包括:

• 日志的路径以及文件名模式. 这里扫描/user下包含深度子目录的所有travel.json文件.
• 分隔符为&#&, 以及对应的域名, 这里是travel_time, from_city, to_city和people.
• 值得注意的是, 也需要在域logSample中填写一个例子.
• 其他需要按照如上的模板填入相关信息. 其中inputType必须是file, inputDetail.logType是delimiter_log.

注意, 多个域的分隔符必须是一致的, 例如下面的文件,期望配置分隔符&|, 并不会起作用.

2017-1-1 10:10:00&nanjing|&shanghai|xiao ming
2017-1-1 20:10:00|beijing&hangzhou|xiao wang

因为这里的文件分隔符并不是一致的, 这种情况就需要用到完整正则表达式的方式了.

时间配置

默认情况下, 日志的时间会使用系统时间, 也就是抓取到这条日志的时间. 如果日志本身包含了更加准确的时间, 例如上面旅行日志的例子, 可以在logtail的配置中指定travel_time为日志时间如下:

{
  "inputDetail": {
    "timeFormat": "%Y-%M-%D %h:%m:s",
    "timeKey": "travel_time"
  }
}

这里指定travel_time为时间关键字, 配置其格式为%Y-%M-%D %h:%m:s, 针对时间格式, 可以进一步参考这里

4. 创建不可见字符的分隔的logtail配置

对于一些不可见字符, 可以在separator域中使用转义:

特殊字符	字符
双引号	"
反斜杠	\ \
斜杠	\/
回退符	\b
换页符	\f
换行符	\n
回车符	\r
横向制表符	\t
纵向制表符	\v
\u 0000	任意字符

例如，这里我们在项目组project1中创建一个分隔符为\u0001的的logtail配置:

> aliyunlog log create_logtail_config --project_name="project1" --config_detail="file://sep_2.json"

文件sep_2.json的内容如下:

{
  "configName": "sep_1",
  "logSample": "2017-1-1 10:10:00\u0001nanjing\u0001shanghai\u0001xiao ming",
  "inputDetail": {
    "logType": "delimiter_log",
    "logPath": "/user",
    "filePattern": "travel.log",
    "separator": "\u0001",
    "key": [
      "travel_time",
      "from_city",
      "to_city",
      "people"
    ]
  },
  "inputType": "file",
  "outputDetail": {
    "logstoreName": "logstore1"
  }
}

注意: 通过CLI配置了不可见字符作为分隔符后，再进入Web控制台编辑这个Logtail配置页面是看不到这些字符，但点击按钮下一步继续其他的配置是不受影响的。

5. 创建正则表达式模式的logtail配置

正则表达式模式利用了强大的正则表达式的解析能力, 提取非结构化文本中的特定模式的域. 适用于文件本身结构较为特殊的情况. 完整的正则表达式可以参考这里

一个典型的例子是NGNIX日志:

10.1.1.1 - - [2018-1-1 10:10:10] "GET / HTTP/1.1" 0.011 180 404 570 "-" "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; 360se)"
10.1.1.1 - - [2018-1-1 10:10:20] "GET / HTTP/1.1" 0.011 180 404 570 "-" "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; 360se)"

可以看到模式是: IP - - [时间] "HTTP请求" 响应时间 请求大小 HTTP状态号 响应大小 "-" "客户端" 这里用对应的正则表达式是: (\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3}) - - \[([^\]+])\] "([\w\s\/\.]+) ([\d\.]+) ([\d\.]+) ([\d\.]+) ([\d\.]+) "-" "[^\"]+"

这里我们在项目组project1中创建一个正则表达式模式的logtail配置:

> aliyunlog log create_logtail_config --project_name="project1" --config_detail="file://reg_1.json"

文件reg_1.json的内容如下:

{
  "configName": "ngnix_1",
  "logSample": "10.1.1.1 - - [13/Mar/2016:10:00:10 +0800] \"GET / HTTP/1.1\" 0.011 180 404 570 \"-\" \"Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; 360se)\"",
  "inputDetail": {
    "logType": "common_reg_log",
    "logPath": "/ngnix",
    "filePattern": "*.log",
    "regex": "(\\d{1,3}\\.\\d{1,3}\\.\\d{1,3}\\.\\d{1,3}) - - \\[([^\\]+])\\] \"([\\w\\s\/\\.]+) ([\\d\\.]+) ([\\d\\.]+) ([\\d\\.]+) ([\\d\\.]+) \"-\" \"[^\\\"]+\"",
    "key": [
      "client_ip",
      "request_time",
      "method_type",
      "process_period",
      "request_bytes",
      "http_status",
      "response_bytes",
      "client_agent"
    ],
    "timeFormat": "%Y-%M-%D %h:%m:%s"
  },
  "inputType": "file",
  "outputDetail": {
    "logstoreName": "logstore1"
  }
}

这里创建了一个属于日志库logstore1的名叫ngnix_1的logtail的配置. 主要的配置包括:

• 日志的路径以及文件名模式. 这里扫描/ngnix下包含深度子目录的所有*.log文件.
• 正则表达式写在regex中, 以及对应的域名放在key中.
• 这里也需要在域logSample中填写一个例子.
• 其他需要按照如上的模板填入相关信息. 其中inputType必须是file, inputDetail.logType是delimiter_log.

时间配置

默认情况下, 日志的时间会使用系统时间, 也就是抓取到这条日志的时间. 如果日志本身包含了更加准确的时间, 例如上面NGNIX日志的例子, 可以在logtail的配置中指定request_time为日志时间如下:

{
  "inputDetail": {
    "timeKey": "request_time",
    "timeFormat": "%Y-%M-%D %h:%m:s"
  }
}

这里通过inputDetail.timeFormat配置其格式为%Y-%M-%D %h:%m:s, 针对时间格式, 可以进一步参考这里

注意: 如果只配置了inputDetail.timeFormat而没有配置inputDetail.timeKey, 默认使用域time.

6. 创建syslog的logtail配置

syslog与文件方式互补, 收集部署上更方便. 除了通过CLI进行配置外, 还需要在logtail客户端通过本地配置文件的方式进行进一步配置. 具体参考这里.

这里我们在项目组project1中创建一个syslog模式的logtail配置:

> aliyunlog log create_logtail_config --project_name="project1" --config_detail="file://syslog_1.json"

文件syslog_1.json的内容如下:

{
  "configName": "syslog_1",
  "inputDetail": {
    "tag": "tag1"
  },
  "inputType": "syslog",
  "outputDetail": {
    "logstoreName": "logstore1"
  }
}

这里创建了一个属于日志库logstore1的名叫syslog_1的logtail的配置. 主要的配置包括:

• 标签tag.
• 其他需要按照如上的模板填入相关信息. 其中inputType必须是syslog, 注意这里没有域inputDetail.logType.

时间配置

要想让syslog配置其作用, 除了通过CLI进行配置外, 还需要在logtail客户端通过本地配置文件的方式进行进一步配置. 具体参考这里. 对于时间的配置, 目前syslog使用系统时间来作为日志的时间.

高级配置项

配置多行日志

很多应用的日志可能偶尔会出现多行的情况, 例如Java的日志会打印一些堆栈信息. 这种情况就需要多行配置. 注意: 目前仅正则表达式模式支持多行配置.

例如:

com.journaldev.log.LoggingExample::main::2017-1-1 01:42:43::Msg977
com.journaldev.log.LoggingExample::main::2017-1-1 03:42:43::Break at
  File "/Applications/PyCharm CE.app/Contents/helpers/pydev/pydevd.py", line 1599, in <module>
    globals = debugger.run(setup['file'], None, None, is_module)
  File "/Applications/PyCharm CE.app/Contents/helpers/pydev/pydevd.py", line 1026, in run
    pydev_imports.execfile(file, globals, locals)  # execute the script
  File "/Users/wjo1212/GithubProjects/aliyun-log-cli/aliyunlogcli/cli.py", line 5, in <module>
    main()
com.journaldev.log.LoggingExample::main::2017-1-1 05:42:43::Msg978

这种情况就需要配置logBeginRegex来指定每一条日志的首行模式:

{
  "inputDetail": {
    "logBeginRegex": "com.journaldev.log.LoggingExample::main::.*"
  }
}

这里配置了inputDetail.logBeginRegex指定了每行新的日志都是以com.journaldev.log.LoggingExample::main::.*作为第一行的.

注意: 这里的正则表达式最后使用了.*对首行进行完整匹配. 这一点是必须的, 例如如下的配置将不起作用:

{
  "inputDetail": {
    "logBeginRegex": "com.journaldev.log.LoggingExample::main::"
  }
}

Logtail客户端的解析配置

Logtail也支持针对特定目标文件, 定义一些高级行为, 所有基于文本的收集方式都支持如下配置:

• 本地缓存 (localStorage):
  • 当日志服务不可用时，日志缓存在机器本地目录，服务恢复后进行续传，默认最大缓存值1GB
  • 默认打开(true)
• 上传原始日志 (enableRawLog)：
  • 开启该功能后默认会新增字段将原始日志内容一并上传
  • 默认关闭 (false)
• Topic生成方式 (topicFormat)：
  • 支持:
    • 空(none): 不配置主题.
    • 机器组(group_topic): 基于所引用的机器组的Topic属性来配置.
    • 基于文件名的正则表达式: 用正则式从路径里提取一部分内容作为Topic。可以用于区分具体用户或实例产生的日志数据。
  • 关于主题, 查考这里
  • 默认是空-不生成主题(none)
• 日志文件编码 (fileEncoding)：
  • 目前支持utf8和gbk
  • 默认是utf8
• 最大监控目录深度 (maxDepth)：
  • 指定从日志源采集日志时，监控目录的最大深度，即最多监控几层日志。最大目录监控深度范围0-1000，0代表只监控本层目录。
  • 默认是100
• 超时属性 (preserve)
  • 如果一个日志文件在指定时间内没有任何更新，则认为该文件已超时。您可以对超时属性指定以下配置。
    • 永不超时(true): 指定持续监控所有日志文件，永不超时。
    • 超时30分钟超时(false): 如日志文件在30分钟内没有更新，则认为已超时，并不再监控该文件。
  • 默认永不超时(true)
• 最大超时目录深度 (preserveDepth)：
  • 当配置为30分钟超时时, 需要配置深度, 范围1-3
• 过滤器配置：
  • 日志只有完全符合过滤器中的条件才会被收集。
  • 分别配置过滤关键字filterKey和对应内容正则表达式filterRegex.

如下示例, 这里关闭了本地存储, 打开了原始文件上传, 配置了一个自定义的基于正则表达式的Topic方式, 文件编码为gbk, 最大目录深度200, 检测文件变化30分钟, 对应深度为3. 同时配置了只抓取关键字from_city和to_city满足特定正则的日志..

{
  "inputDetail": {
    "localStorage": false,
    "enableRawLog": true,
    "topicFormat": "/user.+/(\\w+).log",
    "fileEncoding": "gbk",
    "maxDepth": 200,
    "preserve": false,
    "preserveDepth": 3,
    "filterKey": [
      "from_city",
      "to_city"
    ],
    "filterRegex": [
      "nanjing|shanghai",
      "beijing|hangzhou"
    ]
  }
}

后续操作

完成Logtail的配置后, 还需要应用配置到机器组, 可以通过操作apply_config_to_machine_group来完成.

跨域复制项目配置

介绍

日志服务是在域(Region)范围内以项目(Project)为边界进行操作的, 典型场景下客户会在多个域部署同一套日志项目和日志库来接受日志和查询日志的服务. 因此常常需要跨域复制多个项目的配置. 使用日志服务的CLI可以比较容易的达到这点.

哪些数据需要复制

通常一个日志服务的配置包含以下部分:

• 多个日志库(logstore), 以及其下面的
• 索引配置
• logtail配置

日志库也包含以下配置, 但是一般多个区域内的配置会有所不同, 一般不需要复制.

• 机器组
• 机器组和logtail配置的关联

注意: CLI也支持机器组相关复制, 但需要额外参数打开.

复制项目配置到其他域

这里假设已经完成了CLI的安装.

1. 配置多区域账户

首先CLI中配置多个区域账户, 以便后续操作, 这一步也是一次性的. 如果之前已经做过, 这里可以跳过.

这里配置2个域的账户, 一个杭州公有云, 一个北京公有云.

> aliyunlog configure AKID****123 AKKEY****123 cn-hangzhou.log.aliyuncs.com hz
> aliyunlog configure AKID****123 AKKEY****123 cn-beijing.log.aliyuncs.com bj

注意: 这里的最后一个参数, 用于表明这个账户的名字, 以便后续使用. 不传入默认是main, 也是默认使用的账户.

参考:

• 关于秘钥的配置, 可以参考配置
• 关于日志服务在各个域的Endpoint地址, 可以参考入口服务

2. 复制项目组配置到目标域

假设我们需要将杭州公有云的项目组project1复制到北京公有云, 这可以这样操作:

> aliyunlog log copy_project --from_project="project1" --to_project="project1" --client-name=hz --to_client=bj

这里CLI就会使用账户hz来读取源项目组project1的相关配置, 并使用账户bj来构建一个新的项目组project1. 如果使用默认账户读取读取源项目组, 可以省略参数--client-name

3. 同域项目组配置复制

某些特殊情况, 我们需要在同一个域内复制项目组并保留所有配置项.

这里我们复制账户bj下的project1为project2:

> aliyunlog log copy_project --from_project="project1" --to_project="project2" --copy_machine_group=true --client-name=bj

提示: 传入copy_machine_group为true可以复制机器组的配置以及Logtail配置的管理信息.

拉取日志

介绍

拉取数据(PullLog)指的是针对特定分区, 按照游标来消费日志, 游标指的是服务器接收到日志的时间. 拉取数据的特点是:

1. 没有索引的数据也可以拉取.
2. 按照分区粒度拉取, 每个分区游标互相无关.
3. 速度快, 易并发, 且支持传输压缩.
4. 拉取的数据按照服务器接受的时间排序.

并发下载

命令pull_log_dump针对下载做了优化，直接指定文件路径，即可将日志下载在本地。

aliyunlog log pull_log_dump --project_name="p1" --logstore_name="l1" --from_time="2018-01-24 16:00:00 CST" --to_time="2018-01-24 17:00:00  CST" --file_path="/data/dump_{}.data"

这里拉取从时间2018-01-24 16:00:00 CST到2018-01-24 17:00:00 CST并把数据下载本地，因为是并发，所以会存储在多个文件中，文件名根据/data/dump_{}.data中的{}替换成多个并发的数字（一般是分区号）。文件的内容格式为一行一个日志，例如：

{"k1":"a1", "k2":"a2"}
{"k1":"b1", "k2":"b2"}
...

下载完后会显示下载的文件列表以及日志数，例如：

{"files": {"/data/dump_1.data": 75, "/data/dump_2.data": 14}, "total_count": 89}

时间格式

时间格式推荐是%Y-%m-%d %H:%M:%S %Z, 如2018-01-24 17:00:00 CST, 但也支持其他合法的时间格式, 例如:Jan 01 2018 10:10:10 CST

注意： 时区不仅可以是CST，如果发现拿不到数据，可以改成特定时区例如 UTC-8 或者 UTC+8

时间范围

命令pull_log_dump传入的时间范围, 需要注意几点:

1. 这里的时间指的是服务器接受日志的时间
2. 时间的范围是左闭右开[), 上面例子中16:00:00服务器接受到的日志会被拉取到, 但是17:00:00服务器所接受到的日志不会.

特殊的时间范围

有时我们不关心数据的某一边界, 例如期望获得所有存储的数据到某一天日期位置. 或者获取从某一天开始到目前存储的数据为止的话, 就可以使用特殊游标.

特殊游标包括begin和end. 例如:

aliyunlog log pull_log_dump --project_name="p1" --logstore_name="l1" --shard_id=3 --from_time="begin" --to_time="2018-01-24 17:00:00 CST"

这里拉取所有2018-01-24 17:00:00 CST之前服务器接收到的日志.

又例如:

aliyunlog log pull_log_dump --project_name="p1" --logstore_name="l1" --shard_id=3 --from_time="2018-01-24 17:00:00 CST" --to_time="end"

这里拉取所有2018-01-24 17:00:00 CST开始及之后服务器接收到的日志.

查询日志

介绍

查询数据(GetLog)指的是针对索引数据, 在特定时间范围内搜索特定的日志.

查询日志的特点是:

1. 针对获取索引的数据
2. 按照索引获取, 跨分区, 支持跨多个logstore查询
3. 可以按照主题, 全文分词, 特定索引域等查询条件获取.
4. 按照日志自身时间查询, 而不是服务器接收.

参考查询语法以了解如何打开索引以及查询的相关语法.

查询日志

根据设定好的查询语句查询数据, 即可通过CLI轻松查询数据. 下面例子查询某个时间范围, 某台服务器响应速度大于5秒的访问日志.

aliyunlog log get_log_all --project="p1" --logstore="l1" --query="host:test.com and response_time>5000" --from_time="2018-01-24 16:00:00 CST" --to_time="2018-01-24 17:00:00  CST"

这里拉取从时间2018-01-24 16:00:00 CST到2018-01-24 17:00:00 CST在内满足条件host:test.com and response_time>5000的日志, 例如:

[{"k1":"a1", "k2":"a2"}, {"k1":"b1", "k2":"b2"}, ... ]

注意:

• 这里也可以使用子命令get_log(s)并传入size=-1, 但如果数据量特别多时, 例如总行数100万行的时候, 推荐使用get_log_all.

转换格式并存储到文件

如果期望将数据按照一行一条的形式存储下来, 一般需要加入jmes-filter进行处理. 如果期望存储到文件, 这直接使用>>重定向到文件即可.

例如:

aliyunlog log get_log_all --project="p1" --logstore="l1" --query="host:test.com and response_time>5000" --from_time="2018-01-24 16:00:00 CST" --to_time="2018-01-24 17:00:00  CST" --jmes-filter="join('\n', map(&to_string(@), @))" >> ~/Desktop/test.data

输出将被存储在文件test.data中, 格式为:

{"k1":"a1", "k2":"a2"}
{"k1":"b1", "k2":"b2"}
...

时间格式

时间格式推荐是%Y-%m-%d %H:%M:%S %Z, 如2018-01-24 17:00:00 CST, 但也支持其他合法的时间格式, 例如:Jan 01 2018 10:10:10 CST

注意： 时区不仅可以是CST，如果发现拿不到数据，可以改成特定时区例如 UTC-8 或者 UTC+8

CLI还支持更多其他格式的时间格式，例如2 day ago等，参考这里.

时间范围

命令get_log(s)或get_log_all传入的时间范围, 需要注意几点:

1. 这里的时间指的是解析出的日志时间(日志配置没有指定的情况下, 服务器接收日志的时间会设为这个时间).
2. 时间的范围是左闭右闭[], 上面例子中16:00:00和17:00:00时间的日志都会获得.

跨库查询

使用接口get_project_logs可以跨库查询日志. 例如:

aliyunlog log get_project_logs --request="{\"project\":\"p1\", \"query\":\"select count(1) from logstore1, logstore2, logstore3 where __date__ >'2017-11-10 00:00:00' and __date__ < '2017-11-13 00:00:00'\"}"

具体细节可以参考跨库查询.

使用CLI进行高速跨域日志复制、历史数据重新索引与数仓投递

背景

使用日志服务是不是常常遇到如下烦恼？ 1. 开启了字段索引却无法对历史日志起作用，而手动重建索引又很困难怎么办？ 2. 需要迁移数据，复制数据到其他区域logstore，写代码实现大并发复制又很复杂怎么办？ 3. 投递日志到OSS/ODPS仅仅对新数据起作用，又想投递历史日志怎么办？

现在使用CLI就可以帮你轻松实现这些操作。

介绍

复制数据(CopyData)支持将特定时间范围内的logstore的数据复制到特定logstore中去。其具备如下一些特点:

1. 没有索引的数据也可以同步.
2. 速度快, 易并发, 且支持传输压缩.
3. 拉取的数据按照服务器接受的时间排序.
4. 支持跨域、跨项目库复制。
5. 支持复制数据到同一个logstore（重新索引）。

前提

这里假设已经完成了CLI的安装.

配置多区域账户

首先CLI中配置多个区域账户, 以便后续操作, 这一步也是一次性的. 如果之前已经做过, 这里可以跳过.

这里配置2个域的账户, 一个杭州公有云, 一个北京公有云.

> aliyunlog configure AKID****123 AKKEY****123 cn-hangzhou.log.aliyuncs.com
> aliyunlog configure AKID****123 AKKEY****123 cn-beijing.log.aliyuncs.com bj

注意: 这里的最后一个参数, 仅仅用于表明这个账户的名字, 以便后续使用. 不传入默认是main, 也是默认使用的账户, 这里使用杭州的账户作为默认账户.

参考:

• 关于秘钥的配置, 可以参考配置
• 关于日志服务在各个域的Endpoint地址, 可以参考入口服务

重新索引

如果因为某些特定原因，某个时间范围内的日志没有建立索引，无法被查询和统计。可以如下操作将日志重新写入，就可以实现重新索引的效果。

例如：

aliyunlog log copy_data --project="源project" --logstore="源logstore" --from_time="2018-09-05 0:0:0 CST" --to_time="2018-09-06 0:0:0 CST"

这里将杭州区域的源project的源logstore中服务器在时间范围[“2018-09-05 0:0:0 CST”,”2018-09-06 0:0:0 CST”)内接收到的数据，重新写入到源logstore中去。

注意： 这里仅仅是复制一份数据进入目标logstore，并在写入时自动对其索引，原来的没有被索引的日志依然存在。

跨区域复制数据

有时需要将某一个logstore的日志迁移到另外一个logstore中去时，可以如下操作：

准备好目标logstore

假设目标logstore已经创建好了，并且配置好了索引。这一步操作可以在Web控制台完成，也可以通过CLI的复制logstore配置来完成，或者使用CLI的create_logstore创建日志库，再配置索引，通过命令get_index_config获取索引，调用命令create_index来实现。

复制数据

例如：

aliyunlog log copy_data --project="源项目" --logstore="源logstore" --from_time="2018-09-05 0:0:0 CST" --to_time="2018-09-06 0:0:0 CST" --to_project="目标project" --to_logstore="目标logstore" --to_client="bj"

这里将杭州区域的源project的源logstore中服务器在时间范围[“2018-09-05 0:0:0 CST”,”2018-09-06 0:0:0 CST”)内接收到的数据，写入到北京区域的目标project的目标logstore中去。

注意: 这里用--to_client指定操作目标project的账户为bj，是前面前提中配置的账户名。

投递历史日志到OSS/ODPS

日志服务的投递任务配置好后，仅仅对新接受数据产生作用。这里也可以借助复制数据来实现投递历史日志的效果：

1. 创建一个临时的logstore（不需要配置索引）
2. 在临时logstore上配置投递OSS/ODPS的任务
3. 复制需要投递的日志到目标logstore
4. 投递结束后，删除临时logstore

时间格式

时间格式推荐是%Y-%m-%d %H:%M:%S %Z, 如2018-01-24 17:00:00 CST, 但也支持其他合法的时间格式, 例如:Jan 01 2018 10:10:10 CST

注意： 时区不仅可以是CST，如果发现拿不到数据，可以改成特定时区例如 UTC-8 或者 UTC+8

CLI还支持更多其他格式的时间格式，例如2 day ago等，参考这里.

时间范围

传入的时间范围, 需要注意几点:

1. 这里的时间指的是服务器接受日志的时间
2. 时间的范围是左闭右开[), 上面例子中16:00:00服务器接受到的日志会被拉取到, 但是17:00:00服务器所接受到的日志不会.

特殊的时间范围

有时我们不关心数据的某一边界, 例如期望获得所有存储的数据到某一天日期位置. 或者获取从某一天开始到目前存储的数据为止的话, 就可以使用特殊游标.

特殊游标包括begin和end. 例如:

aliyunlog log copy_data --project="p1" --logstore="l1" --from_time="begin" --to_time="2018-01-24 17:00:00 CST" --to_logstore="l2"

这里复制所有2018-01-24 17:00:00 CST之前服务器接收到的日志到logstorel2。

又例如:

aliyunlog log copy_data --project="p1" --logstore="l1" --from_time="2018-01-24 17:00:00 CST" --to_time="end" --to_logstore="l2"

这里复制所有2018-01-24 17:00:00 CST开始及之后服务器接收到的日志到logstorel2。

使用JMES表达式过滤处理日志

内容敬请期待

日志上传

内容敬请期待

跨域配置索引与报表

如何使用CLI轻松跨域配置索引与报表。

TBD

使用CLI的灵活时间格式来执行查询、报警和下载等

介绍

在查询、统计、拉取时需要传入时间范围，但是经常我们需要传入更灵活的参数，如：”今天凌晨”，”昨天”，”3天前”，”现在”等参数时，大脑需要费力的换算成对应的时间格式，如”2018-8-1 10:10:00”等。 另一方面，用脚本定期跑的时候，需要相对这样的相对值，例如，每5分钟查询，过去60分钟的范围的数据是否有异常等。每次执行希望有一个时间范围是：过去60分钟到现在。

现在这样的方式CLI支持了!

支持的方法

以下方法的参数，如from_time、to_time等目前也接受相对时间了。

方法	作用
get_log	查询统计数据（参数放在文件中），未完成时自动重试。
get_logs	查询统计数据（参数放在命令行中），未完成时自动重试。
get_log_all	迭代式查询更大量的数据，未完成时自动重试。
pull_log	基于服务器接受时间范围，批量拉取特定shard的数据。
pull_log_dump	并发批量下载logstore所有shard的数据到本地。
get_cursor	获取特定服务器时间特定shard的游标（以便基于游标拉取数据）
get_histograms	获取查询范围内的日志分布。

支持的时间格式

除了常规的格式，例如：”%Y-%m-%d %H:%M:%S CST”的例子”2018-01-02 12:12:10 CST”； 也支持”1 hour ago”, “now”, “yesterday 0:0:0”等格式了。更多格式也可以参考https://dateparser.readthedocs.io/en/latest/

相对1分钟: -1min ~ now， 或 1 min ago ~ now

相对4小时: -4h ~ now，或 4 hours ago ~ now 或 4 hour ago

相对4天（24小时）: -1d ~ now，或 1 day ago ~ now

相对1周(7x24小时): -1week ~ now, 或 1 week ago ~ now

今天: today 0:0:0 ~ now

昨天: yesterday 0:0:0 ~ yesterday 23:59:59，或 1 day ago 0:0:0 ~ -1d 23:59:59

前天: the day before yesterday 0:0:0 ~ 2 day ago 23:59:59

例子

实时显示当前网站UV

如下命令

aliyunlog log get_log --project=pdata1 --logstore=log1 --from_time="today 00:00:00" --to_time="now" --query="host: www.dcd.mock-domain.com | select count(distinct remote_addr) as uv" --jmes-filter="join('\n', map(&to_string(uv), @))"

每次运行时都会显示今天的独立访问客户的数量

定期导出客户端IP列表

aliyunlog log get_log --project=pdata1 --logstore=log1 --from_time="yesterday 00:00:00" --to_time="yesterday 23:59:59" --query="host: www.dcd.mock-domain.com | select distinct remote_addr" --jmes-filter="join('\n', map(&to_string(remote_addr), @))"

以上命令可以在每天0:0:1运行，导出昨天的独立访问客户的列表

Elasticsearch 数据迁移

概述

使用日志服务 CLI 可以方便您快速将 Elasticsearch 中的数据导入日志服务。

配置

aliyun-log-cli.readthedocs.io 无法正常显示表格，请参阅tutorial_es_migration_cn.md

数据映射

logstore - index

CLI 默认会将 Elasticsearch index 中的数据迁移至同名的 logstore 中，当然您也可以通过参数 logstore_index_mappings 指定将多个 index 中的数据迁移至一个 logstore。

logstore 不必事先创建，如果 CLI 发现目标 logstore 未创建，会为您在指定的 project 下创建好。

数据类型映射

CLI 会根据 Elasticsearch 的数据类型 在index 对应的 logstore 中创建好索引。

• Core datatypes

Elasticsearch	日志服务
text	text
keyword	text，不分词
long	long
integer	long
short	long
byte	long
double	double
float	double
half_float	double
scaled_float	double
date	text
boolean	text，不分词
binary	n/a
integer_range	json
float_range	json
long_range	json
double_range	json
date_range	json
ip_range	text，不分词

• Complex datatypes

Elasticsearch	日志服务
Array datatype	n/a
Object datatype	json
Nested datatype	n/a

• Geo datatypes

Elasticsearch	日志服务
Geo-point datatype	text
Geo-Shape datatype	text

• Specialised datatypes

Elasticsearch	日志服务
IP datatype	text，不分词
Completion datatype	n/a
Token count datatype	n/a
mapper-murmur3	n/a
Percolator type	n/a
join datatype	n/a

抓取模式

• 为了提高吞吐量，CLI 会为每个 index 的每个 shard 创建一个数据迁移任务，并提交到内部进程池中执行。
• 当全部任务执行完成后，CLI 才会退出。

任务执行情况展示

CLI 默认把任务执行情况存储在~/aliyunlogcli.log中, 也可以配置~/.aliyunlogcli中__loggging__来调整信息存储位置和格式:

[__logging__]
filename=~/es_migration.log
level=info

• 单个迁移任务执行结果展示。

========Tasks Info========
...
task_id=1, slice_id=1, slice_max=10, hosts=localhost:9200, indexes=None, query=None, project=test-project, time_cost_in_seconds=128.71100688, status=CollectionTaskStatus.SUCCESS, count=129330, message=None
...

编号为 1 的迁移任务执行成功，耗时 128.7s，迁移文档数量 129330。

• 迁移任务执行结果汇总信息。

========Summary========
Total started task count: 10
Successful task count: 10
Failed task count: 0
Total collected documentation count: 1000000

MigrationManager 总共启动了 10 个数据数据迁移任务，全部执行成功。迁移文档总数 1000000。

使用样例

• 将 hosts 为 localhost:9200 的 Elasticsearch 中的所有文档导入日志服务的项目 project1 中。

aliyunlog log es_migration --hosts=localhost:9200 --project_name=project1

• 指定将 Elasticsearch 中索引名以 myindex_ 开头的数据写入日志库 logstore1，将索引 index1,index2 中的数据写入日志库 logstore2 中。

aliyunlog log es_migration --hosts=localhost:9200,other_host:9200 --project_name=project1 --logstore_index_mappings='{"logstore1": "myindex_*", "logstore2": "index1,index2"}}'

• 使用参数 query 指定从 Elasticsearch 中抓取 title 字段等于 python 的文档，并使用文档中的字段 date1 作为日志的 time 字段。

aliyunlog log es_migration --hosts=localhost:9200 --project_name=project1 --query='{"query": {"match": {"title": "python"}}}'

• 使用 HTTP 基本认证user:secret@localhost:9200，从 Elasticserch 中迁移数据。

  aliyunlog log es_migration --hosts=user:secret@localhost:9200 --project_name=project1
  

常见问题

Q：是否支持抓取特定时间范围内的 ES 数据？

A：ES 本身并没有内置 time 字段，如果文档中某个字段代表时间，可以使用参数 query 进行过滤。

Q：如何观察迁移任务的执行状况？

A：参考任务执行情况展示部分，将日志级别配置成 info 或以下，并使用命令tail -f <your_log_file>实时观察日志的输出。

Mapped SDK API

Request and Config Class

GetHistogramsRequest([project, logstore, …])	The request used to get histograms of a query from log.
GetLogsRequest([project, logstore, …])	The request used to get logs by a query from log.
GetProjectLogsRequest([project, query])	The request used to get logs by a query from log cross multiple logstores.
ListTopicsRequest([project, logstore, …])	The request used to get topics of a query from log.
ListLogstoresRequest([project])	The request used to list log store from log.
PutLogsRequest([project, logstore, topic, …])	The request used to send data to log.
LogtailConfigGenerator	Generator of Logtial config
PluginConfigDetail(logstoreName, configName, …)	The logtail config for simple mode
SeperatorFileConfigDetail(logstoreName, …)	The logtail config for separator mode
SimpleFileConfigDetail(logstoreName, …[, …])	The logtail config for simple mode
FullRegFileConfigDetail(logstoreName, …[, …])	The logtail config for full regex mode
JsonFileConfigDetail(logstoreName, …[, …])	The logtail config for json mode
ApsaraFileConfigDetail(logstoreName, …[, …])	The logtail config for Apsara mode
SyslogConfigDetail(logstoreName, configName, tag)	The logtail config for syslog mode
MachineGroupDetail([group_name, …])	The machine group detail info
IndexConfig([ttl, line_config, …])	The index config of a logstore
OssShipperConfig(oss_bucket, oss_prefix, …)	A oss ship config
OdpsShipperConfig(odps_endpoint, …[, …])	Odps shipper config
ShipperTask(task_id, task_status, …)	A shipper task

Project

list_project([offset, size])	list the project Unsuccessful opertaion will cause an LogException.
create_project(project_name, project_des)	Create a project Unsuccessful opertaion will cause an LogException.
get_project(project_name)	get project Unsuccessful opertaion will cause an LogException.
delete_project(project_name)	delete project Unsuccessful opertaion will cause an LogException.
copy_project(from_project, to_project[, …])	copy project, logstore, machine group and logtail config to target project, expecting the target project doesn’t contain same named logstores as source project

Logstore

copy_logstore(from_project, from_logstore, …)	copy logstore, index, logtail config to target logstore, machine group are not included yet.
list_logstore(project_name[, …])	list the logstore in a projectListLogStoreResponse Unsuccessful opertaion will cause an LogException.
create_logstore(project_name, logstore_name)	create log store Unsuccessful opertaion will cause an LogException.
get_logstore(project_name, logstore_name)	get the logstore meta info Unsuccessful opertaion will cause an LogException.
update_logstore(project_name, logstore_name)	update the logstore meta info Unsuccessful opertaion will cause an LogException.
delete_logstore(project_name, logstore_name)	delete log store Unsuccessful opertaion will cause an LogException.
list_topics(request)	List all topics in a logstore.

Index

create_index(project_name, logstore_name, …)	create index for a logstore Unsuccessful opertaion will cause an LogException.
update_index(project_name, logstore_name, …)	update index for a logstore Unsuccessful opertaion will cause an LogException.
delete_index(project_name, logstore_name)	delete index of a logstore Unsuccessful opertaion will cause an LogException.
get_index_config(project_name, logstore_name)	get index config detail of a logstore Unsuccessful opertaion will cause an LogException.

Logtail Config

create_logtail_config(project_name, …)	create logtail config in a project Unsuccessful opertaion will cause an LogException.
update_logtail_config(project_name, …)	update logtail config in a project Unsuccessful opertaion will cause an LogException.
delete_logtail_config(project_name, config_name)	delete logtail config in a project Unsuccessful opertaion will cause an LogException.
get_logtail_config(project_name, config_name)	get logtail config in a project Unsuccessful opertaion will cause an LogException.
list_logtail_config(project_name[, offset, size])	list logtail config name in a project Unsuccessful opertaion will cause an LogException.

Machine Group

create_machine_group(project_name, group_detail)	create machine group in a project Unsuccessful opertaion will cause an LogException.
delete_machine_group(project_name, group_name)	delete machine group in a project Unsuccessful opertaion will cause an LogException.
update_machine_group(project_name, group_detail)	update machine group in a project Unsuccessful opertaion will cause an LogException.
get_machine_group(project_name, group_name)	get machine group in a project Unsuccessful opertaion will cause an LogException.
list_machine_group(project_name[, offset, size])	list machine group names in a project Unsuccessful opertaion will cause an LogException.
list_machines(project_name, group_name[, …])	list machines in a machine group Unsuccessful opertaion will cause an LogException.

Apply Logtail Config

apply_config_to_machine_group(project_name, …)	apply a logtail config to a machine group Unsuccessful opertaion will cause an LogException.
remove_config_to_machine_group(project_name, …)	remove a logtail config to a machine group Unsuccessful opertaion will cause an LogException.
get_machine_group_applied_configs(…)	get the logtail config names applied in a machine group Unsuccessful opertaion will cause an LogException.
get_config_applied_machine_groups(…)	get machine group names where the logtail config applies to Unsuccessful opertaion will cause an LogException.

Shard

list_shards(project_name, logstore_name)	list the shard meta of a logstore Unsuccessful opertaion will cause an LogException.
split_shard(project_name, logstore_name, …)	split a readwrite shard into two shards Unsuccessful opertaion will cause an LogException.
merge_shard(project_name, logstore_name, shardId)	split two adjacent readwrite hards into one shards Unsuccessful opertaion will cause an LogException.

Cursor

get_cursor(project_name, logstore_name, …)	Get cursor from log service for batch pull logs Unsuccessful opertaion will cause an LogException.
get_cursor_time(project_name, logstore_name, …)	Get cursor time from log service Unsuccessful opertaion will cause an LogException.
get_previous_cursor_time(project_name, …)	Get previous cursor time from log service.
get_begin_cursor(project_name, …)	Get begin cursor from log service for batch pull logs Unsuccessful opertaion will cause an LogException.
get_end_cursor(project_name, logstore_name, …)	Get end cursor from log service for batch pull logs Unsuccessful opertaion will cause an LogException.

Logs

put_logs(request)	Put logs to log service.
pull_logs(project_name, logstore_name, …)	batch pull log data from log service Unsuccessful opertaion will cause an LogException.
pull_log(project_name, logstore_name, …[, …])	batch pull log data from log service using time-range Unsuccessful opertaion will cause an LogException.
pull_log_dump(project_name, logstore_name, …)	dump all logs seperatedly line into file_path, file_path, the time parameters are log received time on server side.
get_log(project, logstore, from_time, to_time)	Get logs from log service.
get_logs(request)	Get logs from log service.
get_log_all(project, logstore, from_time, …)	Get logs from log service.
get_histograms(request)	Get histograms of requested query from log service.
get_project_logs(request)	Get logs from log service.

Consumer group

create_consumer_group(project, logstore, …)	create consumer group
update_consumer_group(project, logstore, …)	Update consumer group
delete_consumer_group(project, logstore, …)	Delete consumer group
list_consumer_group(project, logstore)	List consumer group
update_check_point(project, logstore, …[, …])	Update check point
get_check_point(project, logstore, …[, shard])	Get check point

Dashboard

create_dashboard(project, detail)	Create Dashboard.
update_dashboard(project, detail)	Update Dashboard.
delete_dashboard(project, entity)	Delete Dashboard.
get_dashboard(project, entity)	Get Dashboard.
list_dashboard(project[, offset, size])	list the Dashboard, get first 100 items by default Unsuccessful opertaion will cause an LogException.

Alert

create_alert(project, detail)	Create Alert.
update_alert(project, detail)	Update Alert.
delete_alert(project, entity)	Delete Alert.
get_alert(project, entity)	Get Alert.
list_alert(project[, offset, size])	list the Alert, get first 100 items by default Unsuccessful opertaion will cause an LogException.

Savedsearch

create_savedsearch(project, detail)	Create Savedsearch.
update_savedsearch(project, detail)	Update Savedsearch.
delete_savedsearch(project, entity)	Delete Savedsearch.
get_savedsearch(project, entity)	Get Savedsearch.
list_savedsearch(project[, offset, size])	list the Savedsearch, get first 100 items by default Unsuccessful opertaion will cause an LogException.

Shipper

create_shipper(project_name, logstore_name, …)	create odps/oss shipper for every type, it only allowed one shipper Unsuccessful opertaion will cause an LogException.
update_shipper(project_name, logstore_name, …)	update odps/oss shipper for every type, it only allowed one shipper Unsuccessful opertaion will cause an LogException.
delete_shipper(project_name, logstore_name, …)	delete odps/oss shipper Unsuccessful opertaion will cause an LogException.
get_shipper_config(project_name, …)	get odps/oss shipper Unsuccessful opertaion will cause an LogException.
list_shipper(project_name, logstore_name)	list odps/oss shipper Unsuccessful opertaion will cause an LogException.
get_shipper_tasks(project_name, …[, …])	get odps/oss shipper tasks in a certain time range Unsuccessful opertaion will cause an LogException.
retry_shipper_tasks(project_name, …)	retry failed tasks , only the failed task can be retried Unsuccessful opertaion will cause an LogException.

ES Migration

MigrationManager([hosts, indexes, query, …])

MigrationManager, migrate data from elasticsearch to aliyun log service

Definitions

class aliyun.log.LogClient(endpoint, accessKeyId, accessKey, securityToken=None, source=None)

Construct the LogClient with endpoint, accessKeyId, accessKey.

Parameters:

• endpoint (string) – log service host name, for example, ch-hangzhou.log.aliyuncs.com or https://cn-beijing.log.aliyuncs.com
• accessKeyId (string) – aliyun accessKeyId
• accessKey (string) – aliyun accessKey

apply_config_to_machine_group(project_name, config_name, group_name)

apply a logtail config to a machine group Unsuccessful opertaion will cause an LogException.

Parameters:

• project_name (string) – the Project name
• config_name (string) – the logtail config name to apply
• group_name (string) – the machine group name

Returns:

ApplyConfigToMachineGroupResponse

Raise:

LogException

arrange_shard(project, logstore, count)

arrange shard to the expected read-write count to a larger one.

Parameters:

• project (string) – project name
• logstore (string) – logstore name
• count (int) – expected read-write shard count. should be larger than the current one.

Returns:

’‘

Raise:

LogException

copy_data(project, logstore, from_time, to_time=None, to_client=None, to_project=None, to_logstore=None, shard_list=None, batch_size=500, compress=True, new_topic=None, new_source=None)

copy data from one logstore to another one (could be the same or in different region), the time is log received time on server side.

Parameters:

• project (string) – project name
• logstore (string) – logstore name
• from_time (string/int) – curosr value, could be begin, timestamp or readable time in readable time like “%Y-%m-%d %H:%M:%S CST” e.g. “2018-01-02 12:12:10 CST”, also support human readable string, e.g. “1 hour ago”, “now”, “yesterday 0:0:0”, refer to https://aliyun-log-cli.readthedocs.io/en/latest/tutorials/tutorial_human_readable_datetime.html
• to_time (string/int) – curosr value, default is “end”, could be begin, timestamp or readable time in readable time like “%Y-%m-%d %H:%M:%S CST” e.g. “2018-01-02 12:12:10 CST”, also support human readable string, e.g. “1 hour ago”, “now”, “yesterday 0:0:0”, refer to https://aliyun-log-cli.readthedocs.io/en/latest/tutorials/tutorial_human_readable_datetime.html
• to_client (LogClient) – logclient instance, if empty will use source client
• to_project (string) – project name, if empty will use source project
• to_logstore (string) – logstore name, if empty will use source logstore
• shard_list (string) – shard number list. could be comma seperated list or range: 1,20,31-40
• batch_size (int) – batch size to fetch the data in each iteration. by default it’s 500
• compress (bool) – if use compression, by default it’s True
• new_topic (string) – overwrite the copied topic with the passed one
• new_source (string) – overwrite the copied source with the passed one

Returns:

LogResponse {“total_count”: 30, “shards”: {0: 10, 1: 20} })

copy_logstore(from_project, from_logstore, to_logstore, to_project=None, to_client=None)

copy logstore, index, logtail config to target logstore, machine group are not included yet. the target logstore will be crated if not existing

Parameters:

• from_project (string) – project name
• from_logstore (string) – logstore name
• to_logstore (string) – target logstore name
• to_project (string) – target project name, copy to same project if not being specified, will try to create it if not being specified
• to_client (LogClient) – logclient instance, use it to operate on the “to_project” if being specified for cross region purpose

Returns:

copy_project(from_project, to_project, to_client=None, copy_machine_group=False)

copy project, logstore, machine group and logtail config to target project, expecting the target project doesn’t contain same named logstores as source project

Parameters:

• from_project (string) – project name
• to_project (string) – project name
• to_client (LogClient) – logclient instance
• copy_machine_group (bool) – if copy machine group resources, False by default.

Returns:

None

create_alert(project, detail)

Create Alert. Unsuccessful opertaion will cause an LogException.

Parameters:

• project (string) – project name
• detail (dict/string) – json string

Returns:

CreateEntityResponse

Raise:

LogException

create_consumer_group(project, logstore, consumer_group, timeout, in_order=False)

create consumer group

Parameters:

• project (string) – project name
• logstore (string) – logstore name
• consumer_group (string) – consumer group name
• timeout (int) – time-out
• in_order (bool) – if consume in order, default is False

Returns:

CreateConsumerGroupResponse

create_dashboard(project, detail)

Create Dashboard. Unsuccessful opertaion will cause an LogException.

Parameters:

• project (string) – project name
• detail (dict/string) – json string

Returns:

CreateEntityResponse

Raise:

LogException

create_external_store(project_name, config)

create log store Unsuccessful opertaion will cause an LogException.

Parameters:project_name (string) – the Project name

:type config : ExternalStoreConfig :param config :external store config

Returns:CreateExternalStoreResponse Raise:LogException

create_index(project_name, logstore_name, index_detail)

create index for a logstore Unsuccessful opertaion will cause an LogException.

Parameters:

• project_name (string) – the Project name
• logstore_name (string) – the logstore name
• index_detail (IndexConfig) – the index config detail used to create index

Returns:

CreateIndexResponse

Raise:

LogException

create_logstore(project_name, logstore_name, ttl=30, shard_count=2, enable_tracking=False, append_meta=False, auto_split=True, max_split_shard=64, preserve_storage=False)

create log store Unsuccessful opertaion will cause an LogException.

Parameters:

• project_name (string) – the Project name
• logstore_name (string) – the logstore name
• ttl (int) – the life cycle of log in the logstore in days, default 30, up to 3650
• shard_count (int) – the shard count of the logstore to create, default 2
• enable_tracking (bool) – enable web tracking, default is False
• append_meta (bool) – allow to append meta info (server received time and IP for external IP to each received log)
• auto_split (bool) – auto split shard, max_split_shard will be 64 by default is True
• max_split_shard (int) – max shard to split, up to 64
• preserve_storage (bool) – if always persist data, TTL will be ignored.

Returns:

CreateLogStoreResponse

Raise:

LogException

create_logtail_config(project_name, config_detail)

create logtail config in a project Unsuccessful opertaion will cause an LogException.

Parameters:

• project_name (string) – the Project name
• config_detail (LogtailConfigGenerator or SeperatorFileConfigDetail or SimpleFileConfigDetail or FullRegFileConfigDetail or JsonFileConfigDetail or ApsaraFileConfigDetail or SyslogConfigDetail or CommonRegLogConfigDetail) – the logtail config detail info, use LogtailConfigGenerator.from_json to generate config: SeperatorFileConfigDetail or SimpleFileConfigDetail or FullRegFileConfigDetail or JsonFileConfigDetail or ApsaraFileConfigDetail or SyslogConfigDetail, Note: CommonRegLogConfigDetail is deprecated.

Returns:

CreateLogtailConfigResponse

Raise:

LogException

create_machine_group(project_name, group_detail)

create machine group in a project Unsuccessful opertaion will cause an LogException.

Parameters:

• project_name (string) – the Project name
• group_detail (MachineGroupDetail) – the machine group detail config

Returns:

CreateMachineGroupResponse

Raise:

LogException

create_project(project_name, project_des)

Create a project Unsuccessful opertaion will cause an LogException.

Parameters:

• project_name (string) – the Project name
• project_des (string) – the description of a project

Returns:

CreateProjectResponse

Raise:

LogException

create_savedsearch(project, detail)

Create Savedsearch. Unsuccessful opertaion will cause an LogException.

Parameters:

• project (string) – project name
• detail (dict/string) – json string

Returns:

CreateEntityResponse

Raise:

LogException

create_shipper(project_name, logstore_name, shipper_name, shipper_type, shipper_config)

create odps/oss shipper for every type, it only allowed one shipper Unsuccessful opertaion will cause an LogException.

Parameters:

• project_name (string) – the Project name
• logstore_name (string) – the logstore name
• shipper_name (string) – the shipper name
• shipper_type (string) – only support “odps” or “oss”
• shipper_config (OssShipperConfig or OdpsShipperConfig) – the detail shipper config, must be OssShipperConfig or OdpsShipperConfig type

Returns:

CreateShipperResponse

Raise:

LogException

delete_alert(project, entity)

Delete Alert. Unsuccessful opertaion will cause an LogException.

Parameters:

• project (string) – project name
• entity (string) – alert name

Returns:

DeleteEntityResponse

Raise:

LogException

delete_consumer_group(project, logstore, consumer_group)

Delete consumer group

Parameters:

• project (string) – project name
• logstore (string) – logstore name
• consumer_group (string) – consumer group name

Returns:

None

delete_dashboard(project, entity)

Delete Dashboard. Unsuccessful opertaion will cause an LogException.

Parameters:

• project (string) – project name
• entity (string) – dashboard name

Returns:

DeleteEntityResponse

Raise:

LogException

delete_external_store(project_name, store_name)

delete log store Unsuccessful opertaion will cause an LogException.

Parameters:

• project_name (string) – the Project name
• store_name (string) – the external store name

Returns:

DeleteExternalStoreResponse

Raise:

LogException

delete_index(project_name, logstore_name)

delete index of a logstore Unsuccessful opertaion will cause an LogException.

Parameters:

• project_name (string) – the Project name
• logstore_name (string) – the logstore name

Returns:

DeleteIndexResponse

Raise:

LogException

delete_logstore(project_name, logstore_name)

delete log store Unsuccessful opertaion will cause an LogException.

Parameters:

• project_name (string) – the Project name
• logstore_name (string) – the logstore name

Returns:

DeleteLogStoreResponse

Raise:

LogException

delete_logtail_config(project_name, config_name)

delete logtail config in a project Unsuccessful opertaion will cause an LogException.

Parameters:

• project_name (string) – the Project name
• config_name (string) – the logtail config name

Returns:

DeleteLogtailConfigResponse

Raise:

LogException

delete_machine_group(project_name, group_name)

delete machine group in a project Unsuccessful opertaion will cause an LogException.

Parameters:

• project_name (string) – the Project name
• group_name (string) – the group name

Returns:

DeleteMachineGroupResponse

Raise:

LogException

delete_project(project_name)

delete project Unsuccessful opertaion will cause an LogException.

Parameters:project_name (string) – the Project name Returns:DeleteProjectResponse Raise:LogException

delete_savedsearch(project, entity)

Delete Savedsearch. Unsuccessful opertaion will cause an LogException.

Parameters:

• project (string) – project name
• entity (string) – savedsearch name

Returns:

DeleteEntityResponse

Raise:

LogException

delete_shard(project_name, logstore_name, shardId)

delete a readonly shard Unsuccessful opertaion will cause an LogException.

Parameters:

• project_name (string) – the Project name
• logstore_name (string) – the logstore name
• shardId (int) – the read only shard id

Returns:

ListShardResponse

Raise:

LogException

delete_shipper(project_name, logstore_name, shipper_name)

delete odps/oss shipper Unsuccessful opertaion will cause an LogException.

Parameters:

• project_name (string) – the Project name
• logstore_name (string) – the logstore name
• shipper_name (string) – the shipper name

Returns:

DeleteShipperResponse

Raise:

LogException

es_migration(hosts, project_name, indexes=None, query=None, scroll='5m', logstore_index_mappings=None, pool_size=10, time_reference=None, source=None, topic=None, wait_time_in_secs=60, auto_creation=True)

migrate data from elasticsearch to aliyun log service

Parameters:

• hosts (string) – a comma-separated list of source ES nodes. e.g. “localhost:9200,other_host:9200”
• project_name (string) – specify the project_name of your log services. e.g. “your_project”
• indexes (string) – a comma-separated list of source index names. e.g. “index1,index2”
• query (string) – used to filter docs, so that you can specify the docs you want to migrate. e.g. ‘{“query”: {“match”: {“title”: “python”}}}’
• scroll (string) – specify how long a consistent view of the index should be maintained for scrolled search. e.g. “5m”
• logstore_index_mappings (string) – specify the mappings of log service logstore and ES index. e.g. ‘{“logstore1”: “my_index*”, “logstore2”: “index1,index2”}, “logstore3”: “index3”}’
• pool_size (int) – specify the size of process pool. e.g. 10
• time_reference (string) – specify what ES doc’s field to use as log’s time field. e.g. “field1”
• source (string) – specify the value of log’s source field. e.g. “your_source”
• topic (string) – specify the value of log’s topic field. e.g. “your_topic”
• wait_time_in_secs (int) – specify the waiting time between initialize aliyun log and executing data migration task. e.g. 60
• auto_creation (bool) – specify whether to let the tool create logstore and index automatically for you. e.g. True

Returns:

MigrationResponse

Raise:

Exception

get_alert(project, entity)

Get Alert. Unsuccessful opertaion will cause an LogException.

Parameters:

• project (string) – project name
• entity (string) – alert name

Returns:

GetEntityResponse

Raise:

LogException

get_begin_cursor(project_name, logstore_name, shard_id)

Get begin cursor from log service for batch pull logs Unsuccessful opertaion will cause an LogException.

Parameters:

• project_name (string) – the Project name
• logstore_name (string) – the logstore name
• shard_id (int) – the shard id

Returns:

GetLogsResponse

Raise:

LogException

get_check_point(project, logstore, consumer_group, shard=-1)

Get check point

Parameters:

• project (string) – project name
• logstore (string) – logstore name
• consumer_group (string) – consumer group name
• shard (int) – shard id

Returns:

ConsumerGroupCheckPointResponse

get_check_point_fixed(project, logstore, consumer_group, shard=-1)

Get check point

Parameters:

• project (string) – project name
• logstore (string) – logstore name
• consumer_group (string) – consumer group name
• shard (int) – shard id

Returns:

ConsumerGroupCheckPointResponse

get_config_applied_machine_groups(project_name, config_name)

get machine group names where the logtail config applies to Unsuccessful opertaion will cause an LogException.

Parameters:

• project_name (string) – the Project name
• config_name (string) – the logtail config name used to apply

Returns:

GetConfigAppliedMachineGroupsResponse

Raise:

LogException

get_cursor(project_name, logstore_name, shard_id, start_time)

Get cursor from log service for batch pull logs Unsuccessful opertaion will cause an LogException.

Parameters:

• project_name (string) – the Project name
• logstore_name (string) – the logstore name
• shard_id (int) – the shard id
• start_time (string/int) – the start time of cursor, e.g 1441093445 or “begin”/”end”, or readable time like “%Y-%m-%d %H:%M:%S CST”, also support human readable string, e.g. “1 hour ago”, “now”, “yesterday 0:0:0”, refer to https://aliyun-log-cli.readthedocs.io/en/latest/tutorials/tutorial_human_readable_datetime.html

Returns:

GetCursorResponse

Raise:

LogException

get_cursor_time(project_name, logstore_name, shard_id, cursor)

Get cursor time from log service Unsuccessful opertaion will cause an LogException.

Parameters:

• project_name (string) – the Project name
• logstore_name (string) – the logstore name
• shard_id (int) – the shard id
• cursor (string) – the cursor to get its service receive time

Returns:

GetCursorTimeResponse

Raise:

LogException

get_dashboard(project, entity)

Get Dashboard. Unsuccessful opertaion will cause an LogException.

Parameters:

• project (string) – project name
• entity (string) – dashboard name

Returns:

GetEntityResponse

Raise:

LogException

get_end_cursor(project_name, logstore_name, shard_id)

Get end cursor from log service for batch pull logs Unsuccessful opertaion will cause an LogException.

Parameters:

• project_name (string) – the Project name
• logstore_name (string) – the logstore name
• shard_id (int) – the shard id

Returns:

GetLogsResponse

Raise:

LogException

get_external_store(project_name, store_name)

get the logstore meta info Unsuccessful opertaion will cause an LogException.

Parameters:

• project_name (string) – the Project name
• store_name (string) – the logstore name

Returns:

GetLogStoreResponse

Raise:

LogException

get_histograms(request)

Get histograms of requested query from log service. Unsuccessful opertaion will cause an LogException.

Parameters:request (GetHistogramsRequest) – the GetHistograms request parameters class. Returns:GetHistogramsResponse Raise:LogException

get_index_config(project_name, logstore_name)

get index config detail of a logstore Unsuccessful opertaion will cause an LogException.

Parameters:

• project_name (string) – the Project name
• logstore_name (string) – the logstore name

Returns:

GetIndexResponse

Raise:

LogException

get_log(project, logstore, from_time, to_time, topic=None, query=None, reverse=False, offset=0, size=100)

Get logs from log service. will retry when incomplete. Unsuccessful opertaion will cause an LogException. Note: for larger volume of data (e.g. > 1 million logs), use get_log_all

Parameters:

• project (string) – project name
• logstore (string) – logstore name
• from_time (int/string) – the begin timestamp or format of time in readable time like “%Y-%m-%d %H:%M:%S CST” e.g. “2018-01-02 12:12:10 CST”, also support human readable string, e.g. “1 hour ago”, “now”, “yesterday 0:0:0”, refer to https://aliyun-log-cli.readthedocs.io/en/latest/tutorials/tutorial_human_readable_datetime.html
• to_time (int/string) – the end timestamp or format of time in readable time like “%Y-%m-%d %H:%M:%S CST” e.g. “2018-01-02 12:12:10 CST”, also support human readable string, e.g. “1 hour ago”, “now”, “yesterday 0:0:0”, refer to https://aliyun-log-cli.readthedocs.io/en/latest/tutorials/tutorial_human_readable_datetime.html
• topic (string) – topic name of logs, could be None
• query (string) – user defined query, could be None
• reverse (bool) – if reverse is set to true, the query will return the latest logs first, default is false
• offset (int) – line offset of return logs
• size (int) – max line number of return logs, -1 means get all

Returns:

GetLogsResponse

Raise:

LogException

get_log_all(project, logstore, from_time, to_time, topic=None, query=None, reverse=False, offset=0)

Get logs from log service. will retry when incomplete. Unsuccessful opertaion will cause an LogException. different with get_log with size=-1, It will try to iteratively fetch all data every 100 items and yield them, in CLI, it could apply jmes filter to each batch and make it possible to fetch larger volume of data.

Parameters:

• project (string) – project name
• logstore (string) – logstore name
• from_time (int/string) – the begin timestamp or format of time in readable time like “%Y-%m-%d %H:%M:%S CST” e.g. “2018-01-02 12:12:10 CST”, also support human readable string, e.g. “1 hour ago”, “now”, “yesterday 0:0:0”, refer to https://aliyun-log-cli.readthedocs.io/en/latest/tutorials/tutorial_human_readable_datetime.html
• to_time (int/string) – the end timestamp or format of time in readable time like “%Y-%m-%d %H:%M:%S CST” e.g. “2018-01-02 12:12:10 CST”, also support human readable string, e.g. “1 hour ago”, “now”, “yesterday 0:0:0”, refer to https://aliyun-log-cli.readthedocs.io/en/latest/tutorials/tutorial_human_readable_datetime.html
• topic (string) – topic name of logs, could be None
• query (string) – user defined query, could be None
• reverse (bool) – if reverse is set to true, the query will return the latest logs first, default is false
• offset (int) – offset to start, by default is 0

Returns:

GetLogsResponse iterator

Raise:

LogException

get_logs(request)

Get logs from log service. Unsuccessful opertaion will cause an LogException. Note: for larger volume of data (e.g. > 1 million logs), use get_log_all

Parameters:request (GetLogsRequest) – the GetLogs request parameters class. Returns:GetLogsResponse Raise:LogException

get_logstore(project_name, logstore_name)

get the logstore meta info Unsuccessful opertaion will cause an LogException.

Parameters:

• project_name (string) – the Project name
• logstore_name (string) – the logstore name

Returns:

GetLogStoreResponse

Raise:

LogException

get_logtail_config(project_name, config_name)

get logtail config in a project Unsuccessful opertaion will cause an LogException.

Parameters:

• project_name (string) – the Project name
• config_name (string) – the logtail config name

Returns:

GetLogtailConfigResponse

Raise:

LogException

get_machine_group(project_name, group_name)

get machine group in a project Unsuccessful opertaion will cause an LogException.

Parameters:

• project_name (string) – the Project name
• group_name (string) – the group name to get

Returns:

GetMachineGroupResponse

Raise:

LogException

get_machine_group_applied_configs(project_name, group_name)

get the logtail config names applied in a machine group Unsuccessful opertaion will cause an LogException.

Parameters:

• project_name (string) – the Project name
• group_name (string) – the group name list

Returns:

GetMachineGroupAppliedConfigResponse

Raise:

LogException

get_previous_cursor_time(project_name, logstore_name, shard_id, cursor, normalize=True)

Get previous cursor time from log service. Note: normalize = true: if the cursor is out of range, it will be nornalized to nearest cursor Unsuccessful opertaion will cause an LogException.

Parameters:

• project_name (string) – the Project name
• logstore_name (string) – the logstore name
• shard_id (int) – the shard id
• cursor (string) – the cursor to get its service receive time
• normalize (bool) – fix the cursor or not if it’s out of scope

Returns:

GetCursorTimeResponse

Raise:

LogException

get_project(project_name)

get project Unsuccessful opertaion will cause an LogException.

Parameters:project_name (string) – the Project name Returns:GetProjectResponse Raise:LogException

get_project_logs(request)

Get logs from log service. Unsuccessful opertaion will cause an LogException.

Parameters:request (GetProjectLogsRequest) – the GetProjectLogs request parameters class. Returns:GetLogsResponse Raise:LogException

get_resource_usage(project)

get resource usage ist the project Unsuccessful opertaion will cause an LogException.

Parameters:client (string) – project name Returns:dict Raise:LogException

get_savedsearch(project, entity)

Get Savedsearch. Unsuccessful opertaion will cause an LogException.

Parameters:

• project (string) – project name
• entity (string) – savedsearch name

Returns:

GetEntityResponse

Raise:

LogException

get_shipper_config(project_name, logstore_name, shipper_name)

get odps/oss shipper Unsuccessful opertaion will cause an LogException.

Parameters:

• project_name (string) – the Project name
• logstore_name (string) – the logstore name
• shipper_name (string) – the shipper name

Returns:

GetShipperConfigResponse

Raise:

LogException

get_shipper_tasks(project_name, logstore_name, shipper_name, start_time, end_time, status_type='', offset=0, size=100)

get odps/oss shipper tasks in a certain time range Unsuccessful opertaion will cause an LogException.

Parameters:

• project_name (string) – the Project name
• logstore_name (string) – the logstore name
• shipper_name (string) – the shipper name
• start_time (int) – the start timestamp
• end_time (int) – the end timestamp
• status_type (string) – support one of [‘’, ‘fail’, ‘success’, ‘running’] , if the status_type = ‘’ , return all kinds of status type
• offset (int) – the begin task offset, -1 means all
• size (int) – the needed tasks count

Returns:

GetShipperTasksResponse

Raise:

LogException

heart_beat(project, logstore, consumer_group, consumer, shards=None)

Heatbeat consumer group

Parameters:

• project (string) – project name
• logstore (string) – logstore name
• consumer_group (string) – consumer group name
• consumer (string) – consumer name
• shards (int list) – shard id list e.g. [0,1,2]

Returns:

None

list_alert(project, offset=0, size=100)

list the Alert, get first 100 items by default Unsuccessful opertaion will cause an LogException.

Parameters:

• project (string) – the Project name
• offset (int) – the offset of all the matched names
• size (int) – the max return names count, -1 means all

Returns:

ListLogStoreResponse

Raise:

LogException

list_consumer_group(project, logstore)

List consumer group

Parameters:

• project (string) – project name
• logstore (string) – logstore name

Returns:

ListConsumerGroupResponse

list_dashboard(project, offset=0, size=100)

list the Dashboard, get first 100 items by default Unsuccessful opertaion will cause an LogException.

Parameters:

• project (string) – the Project name
• offset (int) – the offset of all the matched names
• size (int) – the max return names count, -1 means all

Returns:

ListLogStoreResponse

Raise:

LogException

list_external_store(project_name, external_store_name_pattern=None, offset=0, size=100)

list the logstore in a projectListLogStoreResponse Unsuccessful opertaion will cause an LogException.

Parameters:

• project_name (string) – the Project name
• logstore_name_pattern (string) – the sub name logstore, used for the server to return logstore names contain this sub name
• offset (int) – the offset of all the matched names
• size (int) – the max return names count, -1 means all

Returns:

ListLogStoreResponse

Raise:

LogException

list_logstore(project_name, logstore_name_pattern=None, offset=0, size=100)

list the logstore in a projectListLogStoreResponse Unsuccessful opertaion will cause an LogException.

Parameters:

• project_name (string) – the Project name
• logstore_name_pattern (string) – the sub name logstore, used for the server to return logstore names contain this sub name
• offset (int) – the offset of all the matched names
• size (int) – the max return names count, -1 means all

Returns:

ListLogStoreResponse

Raise:

LogException

list_logstore_acl(project_name, logstore_name, offset=0, size=100)

list acl of a logstore Unsuccessful opertaion will cause an LogException.

Parameters:

• project_name (string) – the Project name
• logstore_name (string) – the logstore name
• offset (int) – the offset of all acl
• size (int) – the max return acl count

Returns:

ListAclResponse

Raise:

LogException

list_logstores(request)

List all logstores of requested project. Unsuccessful opertaion will cause an LogException.

Parameters:request (ListLogstoresRequest) – the ListLogstores request parameters class. Returns:ListLogStoresResponse Raise:LogException

list_logtail_config(project_name, offset=0, size=100)

list logtail config name in a project Unsuccessful opertaion will cause an LogException.

Parameters:

• project_name (string) – the Project name
• offset (int) – the offset of all config names
• size (int) – the max return names count, -1 means all

Returns:

ListLogtailConfigResponse

Raise:

LogException

list_machine_group(project_name, offset=0, size=100)

list machine group names in a project Unsuccessful opertaion will cause an LogException.

Parameters:

• project_name (string) – the Project name
• offset (int) – the offset of all group name
• size (int) – the max return names count, -1 means all

Returns:

ListMachineGroupResponse

Raise:

LogException

list_machines(project_name, group_name, offset=0, size=100)

list machines in a machine group Unsuccessful opertaion will cause an LogException.

Parameters:

• project_name (string) – the Project name
• group_name (string) – the group name to list
• offset (int) – the offset of all group name
• size (int) – the max return names count, -1 means all

Returns:

ListMachinesResponse

Raise:

LogException

list_project(offset=0, size=100)

list the project Unsuccessful opertaion will cause an LogException.

Parameters:

• offset (int) – the offset of all the matched names
• size (int) – the max return names count, -1 means return all data

Returns:

ListProjectResponse

Raise:

LogException

list_project_acl(project_name, offset=0, size=100)

list acl of a project Unsuccessful opertaion will cause an LogException.

Parameters:

• project_name (string) – the Project name
• offset (int) – the offset of all acl
• size (int) – the max return acl count

Returns:

ListAclResponse

Raise:

LogException

list_savedsearch(project, offset=0, size=100)

list the Savedsearch, get first 100 items by default Unsuccessful opertaion will cause an LogException.

Parameters:

• project (string) – the Project name
• offset (int) – the offset of all the matched names
• size (int) – the max return names count, -1 means all

Returns:

ListLogStoreResponse

Raise:

LogException

list_shards(project_name, logstore_name)

list the shard meta of a logstore Unsuccessful opertaion will cause an LogException.

Parameters:

• project_name (string) – the Project name
• logstore_name (string) – the logstore name

Returns:

ListShardResponse

Raise:

LogException

list_shipper(project_name, logstore_name)

list odps/oss shipper Unsuccessful opertaion will cause an LogException.

Parameters:

• project_name (string) – the Project name
• logstore_name (string) – the logstore name

Returns:

ListShipperResponse

Raise:

LogException

list_topics(request)

List all topics in a logstore. Unsuccessful opertaion will cause an LogException.

Parameters:request (ListTopicsRequest) – the ListTopics request parameters class. Returns:ListTopicsResponse Raise:LogException

merge_shard(project_name, logstore_name, shardId)

split two adjacent readwrite hards into one shards Unsuccessful opertaion will cause an LogException.

Parameters:

• project_name (string) – the Project name
• logstore_name (string) – the logstore name
• shardId (int) – the shard id of the left shard, server will determine the right adjacent shardId

Returns:

ListShardResponse

Raise:

LogException

pull_log(project_name, logstore_name, shard_id, from_time, to_time, batch_size=1000, compress=True)

batch pull log data from log service using time-range Unsuccessful opertaion will cause an LogException. the time parameter means the time when server receives the logs

Parameters:

• project_name (string) – the Project name
• logstore_name (string) – the logstore name
• shard_id (int) – the shard id
• from_time (string/int) – curosr value, could be begin, timestamp or readable time in readable time like “%Y-%m-%d %H:%M:%S CST” e.g. “2018-01-02 12:12:10 CST”, also support human readable string, e.g. “1 hour ago”, “now”, “yesterday 0:0:0”, refer to https://aliyun-log-cli.readthedocs.io/en/latest/tutorials/tutorial_human_readable_datetime.html
• to_time (string/int) – curosr value, could be begin, timestamp or readable time in readable time like “%Y-%m-%d %H:%M:%S CST” e.g. “2018-01-02 12:12:10 CST”, also support human readable string, e.g. “1 hour ago”, “now”, “yesterday 0:0:0”, refer to https://aliyun-log-cli.readthedocs.io/en/latest/tutorials/tutorial_human_readable_datetime.html
• batch_size (int) – batch size to fetch the data in each iteration. by default it’s 1000
• compress (bool) – if use compression, by default it’s True

Returns:

PullLogResponse

Raise:

LogException

pull_log_dump(project_name, logstore_name, from_time, to_time, file_path, batch_size=500, compress=True, encodings=None)

dump all logs seperatedly line into file_path, file_path, the time parameters are log received time on server side.

Parameters:

• project_name (string) – the Project name
• logstore_name (string) – the logstore name
• from_time (string/int) – curosr value, could be begin, timestamp or readable time in readable time like “%Y-%m-%d %H:%M:%S CST” e.g. “2018-01-02 12:12:10 CST”, also support human readable string, e.g. “1 hour ago”, “now”, “yesterday 0:0:0”, refer to https://aliyun-log-cli.readthedocs.io/en/latest/tutorials/tutorial_human_readable_datetime.html
• to_time (string/int) – curosr value, could be begin, timestamp or readable time in readable time like “%Y-%m-%d %H:%M:%S CST” e.g. “2018-01-02 12:12:10 CST”, also support human readable string, e.g. “1 hour ago”, “now”, “yesterday 0:0:0”, refer to https://aliyun-log-cli.readthedocs.io/en/latest/tutorials/tutorial_human_readable_datetime.html
• file_path (string) – file path with {} for shard id. e.g. “/data/dump_{}.data”, {} will be replaced with each partition.
• batch_size (int) – batch size to fetch the data in each iteration. by default it’s 500
• compress (bool) – if use compression, by default it’s True
• encodings (string list) – encoding like [“utf8”, “latin1”] etc to dumps the logs in json format to file. default is [“utf8”,]

Returns:

LogResponse {“total_count”: 30, “files”: {‘file_path_1’: 10, “file_path_2”: 20} })

Raise:

LogException

pull_logs(project_name, logstore_name, shard_id, cursor, count=1000, end_cursor=None, compress=True)

batch pull log data from log service Unsuccessful opertaion will cause an LogException.

Parameters:

• project_name (string) – the Project name
• logstore_name (string) – the logstore name
• shard_id (int) – the shard id
• cursor (string) – the start to cursor to get data
• count (int) – the required pull log package count, default 1000 packages
• end_cursor (string) – the end cursor position to get data
• compress (boolean) – if use zip compress for transfer data

Returns:

PullLogResponse

Raise:

LogException

put_log_raw(project, logstore, log_group, compress=None)

Put logs to log service. using raw data in protobuf

Parameters:

• project (string) – the Project name
• logstore (string) – the logstore name
• log_group (LogGroup) – log group structure
• compress (boolean) – compress or not, by default is True

Returns:

PutLogsResponse

Raise:

LogException

put_logs(request)

Put logs to log service. up to 512000 logs up to 10MB size Unsuccessful opertaion will cause an LogException.

Parameters:request (PutLogsRequest) – the PutLogs request parameters class Returns:PutLogsResponse Raise:LogException

remove_config_to_machine_group(project_name, config_name, group_name)

remove a logtail config to a machine group Unsuccessful opertaion will cause an LogException.

Parameters:

• project_name (string) – the Project name
• config_name (string) – the logtail config name to apply
• group_name (string) – the machine group name

Returns:

RemoveConfigToMachineGroupResponse

Raise:

LogException

retry_shipper_tasks(project_name, logstore_name, shipper_name, task_list)

retry failed tasks , only the failed task can be retried Unsuccessful opertaion will cause an LogException.

Parameters:

• project_name (string) – the Project name
• logstore_name (string) – the logstore name
• shipper_name (string) – the shipper name
• task_list (string array) – the failed task_id list, e.g [‘failed_task_id_1’, ‘failed_task_id_2’,…], currently the max retry task count 10 every time

Returns:

RetryShipperTasksResponse

Raise:

LogException

set_source(source)

Set the source of the log client

Parameters:source (string) – new source Returns:None

set_user_agent(user_agent)

set user agent

Parameters:user_agent (string) – user agent Returns:None

split_shard(project_name, logstore_name, shardId, split_hash)

split a readwrite shard into two shards Unsuccessful opertaion will cause an LogException.

Parameters:

• project_name (string) – the Project name
• logstore_name (string) – the logstore name
• shardId (int) – the shard id
• split_hash (string) – the internal hash between the shard begin and end hash

Returns:

ListShardResponse

Raise:

LogException

transform_data(project, logstore, config, from_time, to_time=None, to_client=None, to_project=None, to_logstore=None, shard_list=None, batch_size=500, compress=True, cg_name=None, c_name=None, cg_heartbeat_interval=None, cg_data_fetch_interval=None, cg_in_order=None, cg_worker_pool_size=None)

transform data from one logstore to another one (could be the same or in different region), the time passed is log received time on server side. There’re two mode, batch mode / consumer group mode. For Batch mode, just leave the cg_name and later options as None.

Parameters:

• project (string) – project name
• logstore (string) – logstore name
• config (string) – transform config imported or path of config (in python)
• from_time (string/int) – curosr value, could be begin, timestamp or readable time in readable time like “%Y-%m-%d %H:%M:%S CST” e.g. “2018-01-02 12:12:10 CST”, also support human readable string, e.g. “1 hour ago”, “now”, “yesterday 0:0:0”, refer to https://aliyun-log-cli.readthedocs.io/en/latest/tutorials/tutorial_human_readable_datetime.html
• to_time (string/int) – curosr value, leave it as None if consumer group is configured. could be begin, timestamp or readable time in readable time like “%Y-%m-%d %H:%M:%S CST” e.g. “2018-01-02 12:12:10 CST”, also support human readable string, e.g. “1 hour ago”, “now”, “yesterday 0:0:0”, refer to https://aliyun-log-cli.readthedocs.io/en/latest/tutorials/tutorial_human_readable_datetime.html
• to_client (LogClient) – logclient instance, if empty will use source client
• to_project (string) – project name, if empty will use source project
• to_logstore (string) – logstore name, if empty will use source logstore
• shard_list (string) – shard number list. could be comma seperated list or range: 1,20,31-40
• batch_size (int) – batch size to fetch the data in each iteration. by default it’s 500
• compress (bool) – if use compression, by default it’s True
• cg_name (string) – consumer group name. must configure if it’s consumer group mode.
• c_name (string) – consumer group name for consumer group mode, default: CLI-transform-data-${process_id}
• cg_heartbeat_interval (int) – cg_heartbeat_interval, default 20
• cg_data_fetch_interval (int) – cg_data_fetch_interval, default 2
• cg_in_order (bool) – cg_in_order, default False
• cg_worker_pool_size (int) – cg_worker_pool_size, default 2

Returns:

LogResponse {“total_count”: 30, “shards”: {0: {“count”: 10, “removed”: 1}, 2: {“count”: 20, “removed”: 1}} })

update_alert(project, detail)

Update Alert. Unsuccessful opertaion will cause an LogException.

Parameters:

• project (string) – project name
• detail (dict/string) – json string

Returns:

UpdateEntityResponse

Raise:

LogException

update_check_point(project, logstore, consumer_group, shard, check_point, consumer='', force_success=True)

Update check point

Parameters:

• project (string) – project name
• logstore (string) – logstore name
• consumer_group (string) – consumer group name
• shard (int) – shard id
• check_point (string) – checkpoint name
• consumer (string) – consumer name
• force_success (bool) – if force to succeed

Returns:

None

update_consumer_group(project, logstore, consumer_group, timeout=None, in_order=None)

Update consumer group

Parameters:

• project (string) – project name
• logstore (string) – logstore name
• consumer_group (string) – consumer group name
• timeout (int) – timeout
• in_order (bool) – order

Returns:

None

update_dashboard(project, detail)

Update Dashboard. Unsuccessful opertaion will cause an LogException.

Parameters:

• project (string) – project name
• detail (dict/string) – json string

Returns:

UpdateEntityResponse

Raise:

LogException

update_external_store(project_name, config)

update the logstore meta info Unsuccessful opertaion will cause an LogException.

:param config : external store config

Returns:UpdateExternalStoreResponse Raise:LogException

update_index(project_name, logstore_name, index_detail)

update index for a logstore Unsuccessful opertaion will cause an LogException.

Parameters:

• project_name (string) – the Project name
• logstore_name (string) – the logstore name
• index_detail (IndexConfig) – the index config detail used to update index

Returns:

UpdateIndexResponse

Raise:

LogException

update_logstore(project_name, logstore_name, ttl=None, enable_tracking=None, shard_count=None, append_meta=None, auto_split=None, max_split_shard=None, preserve_storage=None)

update the logstore meta info Unsuccessful opertaion will cause an LogException.

Parameters:

• project_name (string) – the Project name
• logstore_name (string) – the logstore name
• ttl (int) – the life cycle of log in the logstore in days
• enable_tracking (bool) – enable web tracking
• shard_count (int) – deprecated, the shard count could only be updated by split & merge
• append_meta (bool) – allow to append meta info (server received time and IP for external IP to each received log)
• auto_split (bool) – auto split shard, max_split_shard will be 64 by default is True
• max_split_shard (int) – max shard to split, up to 64
• preserve_storage (bool) – if always persist data, TTL will be ignored.

Returns:

UpdateLogStoreResponse

Raise:

LogException

update_logstore_acl(project_name, logstore_name, acl_action, acl_config)

update acl of a logstore Unsuccessful opertaion will cause an LogException.

Parameters:

• project_name (string) – the Project name
• logstore_name (string) – the logstore name
• acl_action (string) – “grant” or “revoke”, grant or revoke the acl_config to/from a logstore
• acl_config (acl_config.AclConfig) – the detail acl config info

Returns:

UpdateAclResponse

Raise:

LogException

update_logtail_config(project_name, config_detail)

update logtail config in a project Unsuccessful opertaion will cause an LogException.

Parameters:

• project_name (string) – the Project name
• config_detail (LogtailConfigGenerator or SeperatorFileConfigDetail or SimpleFileConfigDetail or FullRegFileConfigDetail or JsonFileConfigDetail or ApsaraFileConfigDetail or SyslogConfigDetail or CommonRegLogConfigDetail) – the logtail config detail info, use LogtailConfigGenerator.from_json to generate config: SeperatorFileConfigDetail or SimpleFileConfigDetail or FullRegFileConfigDetail or JsonFileConfigDetail or ApsaraFileConfigDetail or SyslogConfigDetail

Returns:

UpdateLogtailConfigResponse

Raise:

LogException

update_machine_group(project_name, group_detail)

update machine group in a project Unsuccessful opertaion will cause an LogException.

Parameters:

• project_name (string) – the Project name
• group_detail (MachineGroupDetail) – the machine group detail config

Returns:

UpdateMachineGroupResponse

Raise:

LogException

update_project_acl(project_name, acl_action, acl_config)

update acl of a project Unsuccessful opertaion will cause an LogException.

Parameters:

• project_name (string) – the Project name
• acl_action (string) – “grant” or “revoke”, grant or revoke the acl_config to/from a project
• acl_config (acl_config.AclConfig) – the detail acl config info

Returns:

UpdateAclResponse

Raise:

LogException

update_savedsearch(project, detail)

Update Savedsearch. Unsuccessful opertaion will cause an LogException.

Parameters:

• project (string) – project name
• detail (dict/string) – json string

Returns:

UpdateEntityResponse

Raise:

LogException

update_shipper(project_name, logstore_name, shipper_name, shipper_type, shipper_config)

update odps/oss shipper for every type, it only allowed one shipper Unsuccessful opertaion will cause an LogException.

Parameters:

• project_name (string) – the Project name
• logstore_name (string) – the logstore name
• shipper_name (string) – the shipper name
• shipper_type (string) – only support “odps” or “oss” , the type must be same with the oringal shipper
• shipper_config (OssShipperConfig or OdpsShipperConfig) – the detail shipper config, must be OssShipperConfig or OdpsShipperConfig type

Returns:

UpdateShipperResponse

Raise:

LogException

class aliyun.log.LogException(errorCode, errorMessage, requestId='', resp_status=200, resp_header='', resp_body='')

The Exception of the log request & response.

Parameters:

• errorCode (string) – log service error code
• errorMessage (string) – detailed information for the exception
• requestId (string) – the request id of the response, ‘’ is set if client error

class aliyun.log.GetHistogramsRequest(project=None, logstore=None, fromTime=None, toTime=None, topic=None, query=None)

The request used to get histograms of a query from log.

Parameters:

• project (string) – project name
• logstore (string) – logstore name
• fromTime (int/string) – the begin time or format of time in readable time like “%Y-%m-%d %H:%M:%S CST” e.g. “2018-01-02 12:12:10”, also support human readable string, e.g. “1 hour ago”, “now”, “yesterday 0:0:0”, refer to https://aliyun-log-cli.readthedocs.io/en/latest/tutorials/tutorial_human_readable_datetime.html
• toTime (int/string) – the end time or format of time in readable time like “%Y-%m-%d %H:%M:%S CST” e.g. “2018-01-02 12:12:10”, also support human readable string, e.g. “1 hour ago”, “now”, “yesterday 0:0:0”, refer to https://aliyun-log-cli.readthedocs.io/en/latest/tutorials/tutorial_human_readable_datetime.html
• topic (string) – topic name of logs
• query (string) – user defined query

class aliyun.log.GetLogsRequest(project=None, logstore=None, fromTime=None, toTime=None, topic=None, query=None, line=100, offset=0, reverse=False)

The request used to get logs by a query from log.

Parameters:

• project (string) – project name
• logstore (string) – logstore name
• fromTime (int/string) – the begin time, or format of time in format “%Y-%m-%d %H:%M:%S” e.g. “2018-01-02 12:12:10”
• toTime (int/string) – the end time, or format of time in format “%Y-%m-%d %H:%M:%S” e.g. “2018-01-02 12:12:10”
• topic (string) – topic name of logs
• query (string) – user defined query
• line (int) – max line number of return logs
• offset (int) – line offset of return logs
• reverse (bool) – if reverse is set to true, the query will return the latest logs first

class aliyun.log.GetProjectLogsRequest(project=None, query=None)

The request used to get logs by a query from log cross multiple logstores.

Parameters:

• project (string) – project name
• query (string) – user defined query

class aliyun.log.IndexConfig(ttl=1, line_config=None, key_config_list=None, all_keys_config=None, log_reduce=None)

The index config of a logstore

Parameters:

• ttl (int) – this parameter is deprecated, the ttl is same as logstore’s ttl
• line_config (IndexLineConfig) – the index config of the whole log line
• key_config_list (dict) – dict (string => IndexKeyConfig), the index key configs of the keys
• all_keys_config (IndexKeyConfig) – the key config of all keys, the new create logstore should never user this param, it only used to compatible with old config
• log_reduce (bool) – if to enable logreduce

class aliyun.log.ListTopicsRequest(project=None, logstore=None, token=None, line=None)

The request used to get topics of a query from log.

Parameters:

• project (string) – project name
• logstore (string) – logstore name
• token (string) – the start token to list topics
• line (int) – max topic counts to return

class aliyun.log.ListLogstoresRequest(project=None)

The request used to list log store from log.

Parameters:project (string) – project name

class aliyun.log.LogtailConfigGenerator

Generator of Logtial config

class aliyun.log.PluginConfigDetail(logstoreName, configName, plugin, **extended_items)

The logtail config for simple mode

Parameters:

• logstoreName (string) – the logstore name
• configName (string) – the config name
• logPath (string) – folder of log path /apsara/nuwa/
• filePattern (string) – file path, e.g. .log, it will be /apsara/nuwa/…/.log
• localStorage (bool) – if use local cache 1GB when logtail is offline. default is True.
• enableRawLog (bool) – if upload raw data in content, default is False
• topicFormat (string) – “none”, “group_topic” or regex to extract value from file path e.g. “/test/(w+).log” will extract each file as topic, default is “none”
• fileEncoding (string) – “utf8” or “gbk” so far
• maxDepth (int) – max depth of folder to scan, by default its 100, 0 means just scan the root folder
• preserve (bool) – if preserve time-out, by default is False, 30 min time-out if set it as True
• preserveDepth (int) – time-out folder depth. 1-3
• filterKey (string list) – only keep log which match the keys. e.g. [“city”, “location”] will only scan files math the two fields
• filterRegex (string list) – matched value for filterKey, e.g. [“shanghai|beijing|nanjing”, “east”] note, it’s regex value list
• createTime (int) – timestamp of created, only useful when getting data from REST
• modifyTime (int) – timestamp of last modified time, only useful when getting data from REST
• extended_items (dict) – extended items

class aliyun.log.SeperatorFileConfigDetail(logstoreName, configName, logPath, filePattern, logSample, separator, key, timeKey='', timeFormat=None, localStorage=None, enableRawLog=None, topicFormat=None, fileEncoding=None, maxDepth=None, preserve=None, preserveDepth=None, filterKey=None, filterRegex=None, createTime=None, modifyTime=None, **extended_items)

The logtail config for separator mode

Parameters:

• logstoreName (string) – the logstore name
• configName (string) – the config name
• logPath (string) – folder of log path /apsara/nuwa/
• filePattern (string) – file path, e.g. .log, it will be /apsara/nuwa/…/.log
• logSample (string) – log sample. e.g. shanghai|2000|east
• separator (string) – ‘ ‘ for tab, ‘ ‘ for space, ‘|’, up to 3 chars like “&&&” or “||” etc.
• key (string list) – keys to map the fields like [“city”, “population”, “location”]
• timeKey (string) – one key name in key to set the time or set it None to use system time.
• timeFormat (string) – whe timeKey is not None, set its format, refer to https://help.aliyun.com/document_detail/28980.html?spm=5176.2020520112.113.4.2243b18eHkxdNB
• localStorage (bool) – if use local cache 1GB when logtail is offline. default is True.
• enableRawLog (bool) – if upload raw data in content, default is False
• topicFormat (string) – “none”, “group_topic” or regex to extract value from file path e.g. “/test/(w+).log” will extract each file as topic, default is “none”
• fileEncoding (string) – “utf8” or “gbk” so far
• maxDepth (int) – max depth of folder to scan, by default its 100, 0 means just scan the root folder
• preserve (bool) – if preserve time-out, by default is False, 30 min time-out if set it as True
• preserveDepth (int) – time-out folder depth. 1-3
• filterKey (string list) – only keep log which match the keys. e.g. [“city”, “location”] will only scan files math the two fields
• filterRegex (string list) – matched value for filterKey, e.g. [“shanghai|beijing|nanjing”, “east”] note, it’s regex value list
• createTime (int) – timestamp of created, only useful when getting data from REST
• modifyTime (int) – timestamp of last modified time, only useful when getting data from REST
• extended_items (dict) – extended items

class aliyun.log.SimpleFileConfigDetail(logstoreName, configName, logPath, filePattern, localStorage=None, enableRawLog=None, topicFormat=None, fileEncoding=None, maxDepth=None, preserve=None, preserveDepth=None, filterKey=None, filterRegex=None, **extended_items)

The logtail config for simple mode

Parameters:

• logstoreName (string) – the logstore name
• configName (string) – the config name
• logPath (string) – folder of log path /apsara/nuwa/
• filePattern (string) – file path, e.g. .log, it will be /apsara/nuwa/…/.log
• localStorage (bool) – if use local cache 1GB when logtail is offline. default is True.
• enableRawLog (bool) – if upload raw data in content, default is False
• topicFormat (string) – “none”, “group_topic” or regex to extract value from file path e.g. “/test/(w+).log” will extract each file as topic, default is “none”
• fileEncoding (string) – “utf8” or “gbk” so far
• maxDepth (int) – max depth of folder to scan, by default its 100, 0 means just scan the root folder
• preserve (bool) – if preserve time-out, by default is False, 30 min time-out if set it as True
• preserveDepth (int) – time-out folder depth. 1-3
• filterKey (string list) – only keep log which match the keys. e.g. [“city”, “location”] will only scan files math the two fields
• filterRegex (string list) – matched value for filterKey, e.g. [“shanghai|beijing|nanjing”, “east”] note, it’s regex value list
• createTime (int) – timestamp of created, only useful when getting data from REST
• modifyTime (int) – timestamp of last modified time, only useful when getting data from REST
• extended_items (dict) – extended items

class aliyun.log.FullRegFileConfigDetail(logstoreName, configName, logPath, filePattern, logSample, logBeginRegex=None, regex=None, key=None, timeFormat=None, localStorage=None, enableRawLog=None, topicFormat=None, fileEncoding=None, maxDepth=None, preserve=None, preserveDepth=None, filterKey=None, filterRegex=None, **extended_items)

The logtail config for full regex mode

Parameters:

• logstoreName (string) – the logstore name
• configName (string) – the config name
• logPath (string) – folder of log path /apsara/nuwa/
• filePattern (string) – file path, e.g. .log, it will be /apsara/nuwa/…/.log
• logSample (string) – log sample. e.g. shanghai|2000|east
• logBeginRegex (string) – regex to match line, None means ‘.*’, just single line mode.
• regex (string) – regex to extract fields form log. None means (.*), just capture whole line
• key (string list) – keys to map the fields like [“city”, “population”, “location”]. None means [“content”]
• timeFormat (string) – whe timeKey is not None, set its format, refer to https://help.aliyun.com/document_detail/28980.html?spm=5176.2020520112.113.4.2243b18eHkxdNB
• localStorage (bool) – if use local cache 1GB when logtail is offline. default is True.
• enableRawLog (bool) – if upload raw data in content, default is False
• topicFormat (string) – “none”, “group_topic” or regex to extract value from file path e.g. “/test/(w+).log” will extract each file as topic, default is “none”
• fileEncoding (string) – “utf8” or “gbk” so far
• maxDepth (int) – max depth of folder to scan, by default its 100, 0 means just scan the root folder
• preserve (bool) – if preserve time-out, by default is False, 30 min time-out if set it as True
• preserveDepth (int) – time-out folder depth. 1-3
• filterKey (string list) – only keep log which match the keys. e.g. [“city”, “location”] will only scan files math the two fields
• filterRegex (string list) – matched value for filterKey, e.g. [“shanghai|beijing|nanjing”, “east”] note, it’s regex value list
• createTime (int) – timestamp of created, only useful when getting data from REST
• modifyTime (int) – timestamp of last modified time, only useful when getting data from REST
• extended_items (dict) – extended items

class aliyun.log.JsonFileConfigDetail(logstoreName, configName, logPath, filePattern, timeKey='', timeFormat=None, localStorage=None, enableRawLog=None, topicFormat=None, fileEncoding=None, maxDepth=None, preserve=None, preserveDepth=None, filterKey=None, filterRegex=None, createTime=None, modifyTime=None, **extended_items)

The logtail config for json mode

Parameters:

• logstoreName (string) – the logstore name
• configName (string) – the config name
• logPath (string) – folder of log path /apsara/nuwa/
• filePattern (string) – file path, e.g. .log, it will be /apsara/nuwa/…/.log
• timeKey (string) – one key name in key to set the time or set it None to use system time.
• timeFormat (string) – whe timeKey is not None, set its format, refer to https://help.aliyun.com/document_detail/28980.html?spm=5176.2020520112.113.4.2243b18eHkxdNB
• localStorage (bool) – if use local cache 1GB when logtail is offline. default is True.
• enableRawLog (bool) – if upload raw data in content, default is False
• topicFormat (string) – “none”, “group_topic” or regex to extract value from file path e.g. “/test/(w+).log” will extract each file as topic, default is “none”
• fileEncoding (string) – “utf8” or “gbk” so far
• maxDepth (int) – max depth of folder to scan, by default its 100, 0 means just scan the root folder
• preserve (bool) – if preserve time-out, by default is False, 30 min time-out if set it as True
• preserveDepth (int) – time-out folder depth. 1-3
• filterKey (string list) – only keep log which match the keys. e.g. [“city”, “location”] will only scan files math the two fields
• filterRegex (string list) – matched value for filterKey, e.g. [“shanghai|beijing|nanjing”, “east”] note, it’s regex value list
• createTime (int) – timestamp of created, only useful when getting data from REST
• modifyTime (int) – timestamp of last modified time, only useful when getting data from REST
• extended_items (dict) – extended items

class aliyun.log.ApsaraFileConfigDetail(logstoreName, configName, logPath, filePattern, logBeginRegex, localStorage=None, enableRawLog=None, topicFormat=None, fileEncoding=None, maxDepth=None, preserve=None, preserveDepth=None, filterKey=None, filterRegex=None, createTime=None, modifyTime=None, **extended_items)

The logtail config for Apsara mode

Parameters:

• logstoreName (string) – the logstore name
• configName (string) – the config name
• logPath (string) – folder of log path /apsara/nuwa/
• filePattern (string) – file path, e.g. .log, it will be /apsara/nuwa/…/.log
• logBeginRegex (string) – regex to match line, None means ‘.*’, just single line mode.
• localStorage (bool) – if use local cache 1GB when logtail is offline. default is True.
• enableRawLog (bool) – if upload raw data in content, default is False
• topicFormat (string) – “none”, “group_topic” or regex to extract value from file path e.g. “/test/(w+).log” will extract each file as topic, default is “none”
• fileEncoding (string) – “utf8” or “gbk” so far
• maxDepth (int) – max depth of folder to scan, by default its 100, 0 means just scan the root folder
• preserve (bool) – if preserve time-out, by default is False, 30 min time-out if set it as True
• preserveDepth (int) – time-out folder depth. 1-3
• filterKey (string list) – only keep log which match the keys. e.g. [“city”, “location”] will only scan files math the two fields
• filterRegex (string list) – matched value for filterKey, e.g. [“shanghai|beijing|nanjing”, “east”] note, it’s regex value list
• createTime (int) – timestamp of created, only useful when getting data from REST
• modifyTime (int) – timestamp of last modified time, only useful when getting data from REST
• extended_items (dict) – extended items

class aliyun.log.SyslogConfigDetail(logstoreName, configName, tag, localStorage=None, createTime=None, modifyTime=None, **extended_items)

The logtail config for syslog mode

Parameters:

• logstoreName (string) – the logstore name
• configName (string) – the config name
• tag (string) – tag for the log captured
• localStorage (bool) – if use local cache 1GB when logtail is offline. default is True.
• createTime (int) – timestamp of created, only useful when getting data from REST
• modifyTime (int) – timestamp of last modified time, only useful when getting data from REST
• extended_items (dict) – extended items

class aliyun.log.MachineGroupDetail(group_name=None, machine_type=None, machine_list=None, group_type='', group_attribute=None)

The machine group detail info

Parameters:

• group_name (string) – group name
• machine_type (string) – “ip” or “userdefined”
• machine_list (string list) – the list of machine ips or machine userdefined, e.g [“127.0.0.1”, “127.0.0.2”]
• group_type (string) – the machine group type, “” or “Armory”
• group_attribute (dict) – the attributes in group, it contains two optional key : 1. “externalName”: only used if the group_type is “Armory”, its the Armory name 2. “groupTopic”: group topic value

class aliyun.log.PutLogsRequest(project=None, logstore=None, topic=None, source=None, logitems=None, hashKey=None, compress=True, logtags=None)

The request used to send data to log.

Parameters:

• project (string) – project name
• logstore (string) – logstore name
• topic (string) – topic name
• source (string) – source of the logs
• logitems (list<LogItem>) – log data
• hashKey (String) – put data with set hash, the data will be send to shard whose range contains the hashKey
• compress (bool) – if need to compress the logs
• logtags (list) – list of key:value tag pair , [(tag_key_1,tag_value_1) , (tag_key_2,tag_value_2)]

class aliyun.log.OssShipperConfig(oss_bucket, oss_prefix, oss_role_arn, buffer_interval=300, buffer_mb=128, compress_type='snappy')

A oss ship config

Parameters:

• oss_bucket (string) – the oss bucket name
• oss_prefix (string) – the the prefix path where to save the log
• oss_role_arn (string) – the ram arn used to get the temporary write permission to the oss bucket
• buffer_interval (int) – the time(seconds) to buffer before save to oss
• buffer_mb (int) – the data size(MB) to buffer before save to oss
• compress_type (string) – the compress type, only support ‘snappy’ or ‘none’

class aliyun.log.OdpsShipperConfig(odps_endpoint, odps_project, odps_table, log_fields_list, partition_column, partition_time_format, bufferInterval=1800)

Odps shipper config

Parameters:

• odps_endpoint (string) – the odps endpoint
• odps_project (string) – the odps project name
• odps_table (string) – the odps table name
• log_fields_list (string array) – the log field(keys in log) list mapping to the odps table column. e.g log_fields_list=[‘__time__’, ‘key_a’, ‘key_b’], the $log_time, $log_key_a, $log_key_b will mapping to odps table column No.1, No.2, No.3
• partition_column (string array) – the log fields mapping to odps table partition column
• partition_time_format (string) – the time format of __partition_time__, e.g yyyy_MM_dd_HH_mm

class aliyun.log.ShipperTask(task_id, task_status, task_message, task_create_time, task_last_data_receive_time, task_finish_time)

A shipper task

Parameters:

• task_id (string) – the task id
• task_status (string) – one of [‘success’, ‘running’, ‘fail’]
• task_message (string) – the error message of task_status is ‘fail’
• task_create_time (int) – the task create time (timestamp from 1970.1.1)
• task_last_data_receive_time (int) – last log data receive time (timestamp)
• task_finish_time (int) – the task finish time (timestamp)

class aliyun.log.es_migration.MigrationManager(hosts=None, indexes=None, query=None, scroll='5m', endpoint=None, project_name=None, access_key_id=None, access_key=None, logstore_index_mappings=None, pool_size=10, time_reference=None, source=None, topic=None, wait_time_in_secs=60, auto_creation=True)

MigrationManager, migrate data from elasticsearch to aliyun log service

Parameters:

• hosts (string) – a comma-separated list of source ES nodes. e.g. “localhost:9200,other_host:9200”
• indexes (string) – a comma-separated list of source index names. e.g. “index1,index2”
• query (string) – used to filter docs, so that you can specify the docs you want to migrate. e.g. ‘{“query”: {“match”: {“title”: “python”}}}’
• scroll (string) – specify how long a consistent view of the index should be maintained for scrolled search. e.g. “5m”
• endpoint (string) – specify the endpoint of your log services. e.g. “cn-beijing.log.aliyuncs.com”
• project_name (string) – specify the project_name of your log services. e.g. “your_project”
• access_key_id (string) – specify the access_key_id of your account.
• access_key (string) – specify the access_key of your account.
• logstore_index_mappings (string) – specify the mappings of log service logstore and ES index. e.g. ‘{“logstore1”: “my_index*”, “logstore2”: “index1,index2”}, “logstore3”: “index3”}’
• pool_size (int) – specify the size of process pool. e.g. 10
• time_reference (string) – specify what ES doc’s field to use as log’s time field. e.g. “field1”
• source (string) – specify the value of log’s source field. e.g. “your_source”
• topic (string) – specify the value of log’s topic field. e.g. “your_topic”
• wait_time_in_secs (int) – specify the waiting time between initialize aliyun log and executing data migration task. e.g. 60
• auto_creation (bool) – specify whether to let the tool create logstore and index automatically for you. e.g. True

Indices and tables

• Index
• Search Page