Aliyun Log CLI¶
User Guide¶
Content¶
Introduction¶
The Alicloud log service provides with Web and SDK flavor to operate log service and analyzie logs. To make it more convinient to do automation, we release this command line interface (CLI).
Brief¶
Alicloud log service command line console, support almost all operations as web. It also supports incomplete log query check and query cross multiple pages. It could even do project settings copy cross multiple regions.
Major Features¶
- Support almost all 50+ REST API of log service.
- Multiple account support to support cross region operation.
- Log query incomplete check and automatically query cross pagination.
- Multiple confidential storage types, from file, commandline to env variables.
- Support command line based or file based inputs, complete formation validations.
- Support JMES filter to do further process on results, e.g. select specific fields from json.
- Cross platforms support (Windows, Linux and Mac), Python based and friendly to Py2 and Py3 even Pypy. Support Pip installation.
Installation¶
Supported Version¶
Python 2.6, 2.7, 3.3, 3.4, 3.5, 3.6, PyPy, PyPy3
Installation Method¶
Run below command to install the CLI:
> pip install -U aliyun-log-cli
Note
On mac it’s recommended to use pip3 to install the CLI.
> brew install python3
> pip3 install -U aliyun-log-cli
if you encounter errors like
OSError: [Errno 1] Operation not permitted
, try to use option
--user
to install:
> pip3 install -U aliyun-log-cli --user
Alicloud ECS which may have limited internet access
You could try the mirrors of local network provider, for Alicloud ECS, you can try below noe:
pip/pip3 install -U aliyun-log-cli --index http://mirrors.aliyun.com/pypi/simple/ --trusted-host mirrors.aliyun.com
Offline Installation¶
Since 0.1.12, we provide offline package for mac x64 and linux x64 platform.
Follow below ways to install it. 1. download the package from release
page 2. unzip it
to a local folder, like cli_packages
, you can see some whl files
inside it. 3. if you don’t have pip, install pip first:
python pip-10.0.1-py2.py3-none-any.whl/pip install --no-index cli_packages/pip-10.0.1-py2.py3-none-any.whl
- install the CLI:
pip install aliyun-log-cli --no-index --find-links=cli_packages
verify it
> aliyunlog --version
FAQ of Installation¶
- Encoutering errr
TLSV1_ALERT_PROTOCOL_VERSION
when installing CLI:
> pip install aliyun-log-cli
Collecting aliyun-log-cli
Could not fetch URL https://pypi.python.org/simple/aliyun-log-cli/: There was a problem confirming the ssl certificate: [SSL: TLSV1_ALERT_PROTOCOL_VERSION] tlsv1 alert protocol version (_ssl.c:590) - skipping
Could not find a version that satisfies the requirement aliyun-log-cli (from versions: )
No matching distribution found for aliyun-log-cli
Solution: Please upgrade pip and retry:
pip install pip -U
- On Linux/Mac, cannot find command
aliyunlog
?
it’s caused by the missing of shell of aliyunlog
, you could make one
by yourself.
2.1. find python path:
for linux or mac:
which python
2.2. create a shell script named aliyunlog
with below content and
allow to execute it. And put it into path folder:
#!<python path here with ! ahead>
import re
import sys
from pkg_resources import load_entry_point
if __name__ == '__main__':
sys.argv[0] = re.sub(r'(-script\.pyw?|\.exe)?$', '', sys.argv[0])
sys.exit(
load_entry_point('aliyun-log-cli', 'console_scripts', 'aliyunlog')()
)
for linux or mac, it could be put under /usr/bin/
.
2.3. verify it
> aliyunlog --version
- Fail to install module
regex
?
Refer to below link to install python-devel via yun
, apt-get
or
manually.
https://rpmfind.net/linux/rpm2html/search.php?query=python-devel
Full Usage list¶
Run below command to get the full usage list:
> aliyunlog --help
Note: aliyun
command is deprecated to prevent conflict with
Universal Ali-cloud CLI.
it will show the full usage.
Note aliyunlog
is recommended in case the aliyun
conflict
with others.
Configure CLI¶
Refer to Configure CLI.
Input and Output¶
Inputs¶
- Normally case:
> aliyunlog log get_logs --request="{\"topic\": \"\", \"logstore\": \"logstore1\", \"project\": \"dlq-test-cli-123\", \"toTime\": \"2018-01-01 10:10:10\", \"offset\": \"0\", \"query\": \"*\", \"line\": \"10\", \"fromTime\": \"2018-01-01 08:08:08\", \"reverse\":\"false\"}"
- Input via file: You could store the content of one parameter into a
file and pass it via the command line with prefix
file://
:
> aliyunlog log get_logs --request="file://./get_logs.json"
the content in file get_logs.json
as below. Note: the \
is
unnecessary to escape the “.
{
"topic": "",
"logstore": "logstore1",
"project": "project1",
"toTime": "2018-01-01 11:11:11",
"offset": "0",
"query": "*",
"line": "10",
"fromTime": "2018-01-01 10:10:10",
"reverse": "true"
}
Parameter Validation¶
- Mandatory check: if one mandatory parameter is missed, it will report error with usage info.
- Format of parameter’s value will be validated. e.g. int, bool, string list, special data structure.
- for boolean, it support:
- true (case insensitive), T, 1
- false (case insensitive), F, 0
- String list support as [“s1”, “s2”]
Output¶
- For operations like Create, Update and Delete, there’s no output except the exit code is 0 which means success.
- For operations like Get and List, it will output in json format.
- For errors, it will report in json format as below:
{
"errorCode":"...",
"errorMessage":"..."
}
Filter output¶
It’s supported to filter output via JMES:
Examples:
> aliyunlog log get_logs ...
which outputs:
[ {"__source__": "ip1", "key": "log1"}, {"__source__": "ip2", "key": "log2"} ]
You could use below --jmes-filter
to break log into each line:
> aliyunlog log get_logs ... --jmes-filter="join('\n', map(&to_string(@), @))"
output:
{"__source__": "ip1", "key": "log1"}
{"__source__": "ip2", "key": "log2"}
Further Process¶
You could use >>
to store the output to a file. or you may want to
process the output using your own cmd. For example, there’s another way
to if you may want to break the logs into each line. you could append
thd command with a |
on linux/unix:
| python2 -c "from __future__ import print_function;import json;map(lambda x: print(json.dumps(x).encode('utf8')), json.loads(raw_input()));"
or
| python3 -c "import json;list(map(lambda x: print(json.dumps(x)), json.loads(input())));"
e.g.
aliyunlog log get_log .... | | python2 -c "from __future__ import print_function;import json;map(lambda x: print(json.dumps(x).encode('utf8')), json.loads(raw_input()));" >> data.txt
Command Reference¶
Command Specification¶
1. aliyunlog log <subcommand> [parameters | global options]
2. aliyunlog configure <access_id> <access-key> <endpoint>
3. aliyunlog [--help | --version]
Alias¶
There’s also an alias aliyunlog
for the CLI in case the command
aliyun
conflicts with others.
1. aliyunlog log <subcommand> [parameters | global options]
2. aliyunlog configure <access_id> <access-key> <endpoint>
3. aliyunlog [--help | --version]
Subcommand and parameters¶
Actually, the CLI leverage aliyun-log-python-sdk
, which maps the
command into the methods of aliyun.log.LogClient
. The parameters of
command line is mapped to the parameters of methods. For the detail spec
of parameters, please refer to the Mapped Python SDK API
Spec
Examples:
def create_logstore(self, project_name, logstore_name, ttl=2, shard_count=30):
Mapped to CLI:
> aliyunlog log create_logstore
--project_name=<value>
--logstore_name=<value>
[--ttl=<value>]
[--shard_count=<value>]
Global options¶
All the commands support below optional global options:
[--access-id=<value>]
[--access-key=<value>]
[--region-endpoint=<value>]
[--client-name=<value>]
[--jmes-filter=<value>]
Command categories¶
- Project management
- Logstore management
- Shard management
- Machine group management
- Logtail config management
- Machine group and Logtail Config Mapping
- Index management
- Cursor management
- Logs write and consume
- Shipper management
- Consumer group management
- Elasticsearch data migration
Project management
Project management
- list_project
- create_project
- get_project
- delete_project
- copy_project
- copy all configurations including logstore, logtail, and index config from project to another project which could be in different region.
> aliyunlog log copy_project --from_project="p1" --to_project="p1" --to_client="account2"
- Note:
to_client
is another account configured viaaliyunlog configure
, it’s OK to passmain
or not to copy inside the same region. - Refer to Copy project settings cross regions to learn more.
Logstore management
Logstore management
- create_logstore
- delete_logstore
- get_logstore
- update_logstore
- list_logstore
Shard management
Shard management
- list_shards
- split_shard
- merge_shard
Machine group management
Machine group management
- create_machine_group
- Format of partial parameter:
{
"machine_list": [
"machine1",
"machine2"
],
"machine_type": "userdefined",
"group_name": "group_name2",
"group_type": "Armory",
"group_attribute": {
"externalName": "ex name",
"groupTopic": "topic x"
}
}
- delete_machine_group
- update_machine_group
- get_machine_group
- list_machine_group
- list_machines
Logtail config management
Logtail config management
- create_logtail_config
- 参考Create Logtail Configuration了解如何创建各种格式的Logtail配置.
- update_logtail_config
- delete_logtail_config
- get_logtail_config
- list_logtail_config
Machine group and Logtail Config Mapping
Machine group and Logtail Config Mapping
- apply_config_to_machine_group
- remove_config_to_machine_group
- get_machine_group_applied_configs
- get_config_applied_machine_groups
Index management
Index management
- create_index
- Format of partial parameter:
{
"keys": {
"f1": {
"caseSensitive": false,
"token": [
",",
" ",
"\"",
"\"",
";",
"=",
"(",
")",
"[",
"]",
"{",
"}",
"?",
"@",
"&",
"<",
">",
"/",
":",
"\n",
"\t"
],
"type": "text",
"doc_value": true
},
"f2": {
"doc_value": true,
"type": "long"
}
},
"storage": "pg",
"ttl": 2,
"index_mode": "v2",
"line": {
"caseSensitive": false,
"token": [
",",
" ",
"\"",
"\"",
";",
"=",
"(",
")",
"[",
"]",
"{",
"}",
"?",
"@",
"&",
"<",
">",
"/",
":",
"\n",
"\t"
]
}
}
- update_index
- delete_index
- get_index_config
- list_topics
Cursor management
Cursor management
- get_cursor
- get_cursor_time
- get_previous_cursor_time
- get_begin_cursor
- get_end_cursor
Logs write and consume
Logs write and consume
- put_logs
- Format of parameter:
{
"project": "dlq-test-cli-35144",
"logstore": "logstore1",
"topic": "topic1",
"source": "source1",
"logtags": [
[
"tag1",
"v1"
],
[
"tag2",
"v2"
]
],
"hashKey": "1231231234",
"logitems": [
{
"timestamp": 1510579341,
"contents": [
[
"key1",
"v1"
],
[
"key2",
"v2"
]
]
},
{
"timestamp": 1510579341,
"contents": [
[
"key3",
"v3"
],
[
"key4",
"v4"
]
]
}
]
}
- get_logs
- Format of parameter:
{
"topic": "",
"logstore": "logstore1",
"project": "dlq-test-cli-35144",
"toTime": "2018-01-01 11:11:11",
"offset": "0",
"query": "*",
"line": "10",
"fromTime": "2018-01-01 10:10:10",
"reverse": "true"
}
- It will fetch all data when
line
is passed as -1. But if have large volume of data exceeding 1GB, better to useget_log_all
- get_log_all
- this API is similar as
get_logs
, but it will fetch data iteratively and output them by chunk. It’s used for large volume of data fetching. - get_histograms
- pull_logs
- pull_log
- this API is similar as
pull_logs
, but it allow readable parameter and allow to fetch data iteratively and output them by chunk. It’s used for large volume of data fetching. - pull_log_dump
- this API will dump data from all shards to local files concurrently.
Shipper management
Shipper management
- create_shipper
- Format of partial parameter:
{
"oss_bucket": "dlq-oss-test1",
"oss_prefix": "sls",
"oss_role_arn": "acs:ram::1234:role/aliyunlogdefaultrole",
"buffer_interval": 300,
"buffer_mb": 128,
"compress_type": "snappy"
}
- update_shipper
- delete_shipper
- get_shipper_config
- list_shipper
- get_shipper_tasks
- retry_shipper_tasks
Consumer group management
Consumer group management
- create_consumer_group
- update_consumer_group
- delete_consumer_group
- list_consumer_group
- update_check_point
- get_check_point
Elasticsearch data migration
Elasticsearch data migration
- migrate
- please refer to Elasticsearch data migration to know how to migrate data from elasticsearch to aliyun log service.
Best Practice¶
Troubleshooting¶
By default, CLI store erros or warnings at ~/aliyunlogcli.log
, it’s
also configurable via file ~/.aliyunlogcli, section __loggging__
to
adjust the logging level and location:
[__logging__]
filename= # default: ~/aliyunlogcli.log, Rotated when hit filebytes
filebytes= # Deafult: 5242880 (5MB), file size of each log before rotation, Unit: Bytes
backupcount= # Default: 5, file backup file
#filemode= # deprecated
format= # default: %(asctime)s %(levelname)s %(filename)s:%(lineno)d %(funcName)s %(message)s
datefmt= # default: "%Y-%m-%d %H:%M:%S", could be strftime() compitable date/time formatting string
level= # default: warn, could be: info, error, fatal, critical, debug
Other resources¶
- Alicloud Log Service homepage:https://www.alibabacloud.com/product/log-service
- Alicloud Log Service doc:https://www.alibabacloud.com/help/product/28958.htm
- Alicloud Log Python SDK doc: http://aliyun-log-python-sdk.readthedocs.io/
- for any issues, please submit support tickets
用户手册¶
内容¶
简介
日志服务(SLS)提供了Web和SDK的方式进行日志采集、管理、查询和分析。为了满足越来越多的的自动化日志配置查询分析的需求,现重磅发布SLS的命令行工具(Command Line Interface - CLI)。
基本介绍
日志服务命令行工具CLI支持几乎所有操作,日志查询支持完整性检查与自动分页、支持多账户与跨域复制。
主要功能
- 支持大部分的日志服务REST接口。
- 支持多账号,方便测试与跨域操作。
- 日志查询完整性查询与自动分页。
- 支持多种方式存储和配置秘钥,适用于各种情况。
- 支持通过命令行或者文件的形式输入复杂参数,并校验参数内容格式。
- 支持JMES过滤器对结果进行进一步处理,方便选择特定信息。
- 跨平台(Windows/Linux和Mac)支持,基于Python(Py2.6+和Py3.3+平台友好),支持Pip安装。
安装
操作系统
日志服务CLI支持以下操作系统:
- Windows
- Mac OS
- Linux
支持版本
Python 2.6、2.7、3.3、3.4、3.5、3.6、PyPy和PyPy3。
安装方式
执行以下命令安装日志服务CLI。
> pip install -U aliyun-log-cli
注意
Mac上推荐使用pip3安装CLI,首选需要安装Python3:
> brew install python3
> pip3 install -U aliyun-log-cli
Mac上如果你安装时遇到了权限相关的错误,如:OSError: [Errno 1] Operation not permitted
,尝试使用如下命令安装:
> pip3 install -U aliyun-log-cli --user
外网访问受限的阿里云ECS
因为安全策略等原因,某些云服务器可能无法访问Pypi服务器,可以尝试使用本地Pypi镜像,例如阿里云服务器可以尝试:
pip/pip3 install -U aliyun-log-cli --index http://mirrors.aliyun.com/pypi/simple/ --trusted-host mirrors.aliyun.com
Note:也可以使用清华的索引-i https://pypi.tuna.tsinghua.edu.cn/simple
离线安装
从 0.1.12开始, 我们为Linux x64和MacOS x64提供离线的安装包.
如下步骤安装:
- 从release页面下载对应离线安装包.
- 解压到本地目录, 例如
cli_packages
, 可以看到许多whl文件在里面. - 如果还没有
pip
, 可以如下安装:
python pip-10.0.1-py2.py3-none-any.whl/pip install --no-index cli_packages/pip-10.0.1-py2.py3-none-any.whl
- 安装CLI:
pip install aliyun-log-cli --no-index --find-links=cli_packages
验证:
> aliyunlog --version
常见安装问题
- 安装时报TLSV1_ALERT_PROTOCOL_VERSION错误
> pip install aliyun-log-cli
Collecting aliyun-log-cli
Could not fetch URL https://pypi.python.org/simple/aliyun-log-cli/: There was a problem confirming the ssl certificate: [SSL: TLSV1_ALERT_PROTOCOL_VERSION] tlsv1 alert protocol version (_ssl.c:590) - skipping
Could not find a version that satisfies the requirement aliyun-log-cli (from versions: )
No matching distribution found for aliyun-log-cli
解答: 请先升级pip:
pip install pip -U
- 找不到命令
aliyunlog
?
Linux/Mac下, 因为某种原因创建脚本aliyunlog
链接时未成功,
可以手工创建一个, 如下:
2.1. 找到Python路径:
在linux或mac, 执行命令可以看到:
which python
2.2. 创建一个叫做aliyunlog
文件, 赋予执行权限, 内容如下,
并放到PATH目录下 :
#!<python路径放这里,注意有一个感叹号!>
import re
import sys
from pkg_resources import load_entry_point
if __name__ == '__main__':
sys.argv[0] = re.sub(r'(-script\.pyw?|\.exe)?$', '', sys.argv[0])
sys.exit(
load_entry_point('aliyun-log-cli', 'console_scripts', 'aliyunlog')()
)
对于linux或mac, 可以考虑放到/usr/bin/
目录下
2.3. 验证, 执行如试下命令看一下是否成功. ```shell > aliyunlog –version
- Regex模块安装失败? 如果存在安装Regex失败的错误,
可以参考使用
yun
/apt-get
或者手动安装一下python-devel https://rpmfind.net/linux/rpm2html/search.php?query=python-devel
完整参数列表
执行以下命令查看日志服务CLI参数列表。
> aliyunlog --help
注意: 命令aliyun
已经废弃(为了避免与通用阿里云CLI冲突).
会显示完整命令。
Note
aliyunlog
和aliyun
都可以,推荐使用aliyunlog
以防冲突。
配置说明
参考配置CLI
输入输出
输入
- 一般输入
> aliyunlog log get_logs --request="{\"topic\": \"\", \"logstore\": \"logstore1\", \"project\": \"dlq-test-cli-123\", \"toTime\": \"2018-01-01 11:11:11\", \"offset\": \"0\", \"query\": \"*\", \"line\": \"10\", \"fromTime\": \"2018-01-01 10:10:10\", \"reverse\":\"false\"}"
- 文件输入
也可以将上面参数放到一个文件里面, 简化命令行,
需要义file://
开头+文件路径即可:
> aliyunlog log get_logs --request="file://./get_logs.json"
文件get_logs.json
内容如下, 注意:
文件中不需要反斜杠\
来转义.
{
"topic": "",
"logstore": "logstore1",
"project": "project1",
"toTime": "2018-01-01 11:11:11",
"offset": "0",
"query": "*",
"line": "10",
"fromTime": "2018-01-01 10:10:10",
"reverse": "true"
}
参数校验
- 必填的参数没有填写时会报错, 输出参数列表
- 参数格式本身会进行校验, 例如int, bool, string list, 特定数据结构等
- bool支持的形式有:
- true (大小写不敏感), T, 1
- false (大小写不敏感), F, 0
- 字符串列表支持的形式为[“s1”, “s2”]
输出
- 对于Create, Update, Delete操作, 一般脚本无输出, exit code=0表示成功.
- 对于Get/List操作, 以json格式输出内容
- 错误情况下, 以如下格式返回错误:
{
"errorCode":"...",
"errorMessage":"..."
}
输出过滤
支持通过JMES过滤输出的结果.
例如:
> aliyunlog log get_logs ...
以上命令的输出是:
[ {"__source__": "ip1", "key": "log1"}, {"__source__": "ip2", "key": "log2"} ]
通过以下命令将日志分隔为每一行:
> aliyunlog log get_logs ... --jmes-filter="join('\n', map(&to_string(@), @))"
输出:
{"__source__": "ip1", "key": "log1"}
{"__source__": "ip2", "key": "log2"}
进一步处理
你可以使用>>
来讲输出存储到一个文件. 某些时候,
你需要使用其他命令进行处理, 例如,
这里介绍另一个把json格式的日志分行打印的方法. 在Linux/Unix下,
你可以在命令后通过添加一个|
来进一步处理.
| python2 -c "from __future__ import print_function;import json;map(lambda x: print(json.dumps(x).encode('utf8')), json.loads(raw_input()));"
or
| python3 -c "import json;list(map(lambda x: print(json.dumps(x)), json.loads(input())));"
例如:
aliyunlog log get_log .... | python2 -c "from __future__ import print_function;import json;map(lambda x: print(json.dumps(x).encode('utf8')), json.loads(raw_input()));" >> data.txt
命令参考
命令规范
1. aliyunlog log <subcommand> [parameters | global options]
2. aliyunlog configure <access_id> <access-key> <endpoint> [<client-name>]
3. aliyunlog [--help | --version]
命令别名
日志服务的CLI也有一个别名aliyunlog
,
如果aliyun
这个命令冲突了, 可以尝试使用aliyunlog
:
1. aliyunlog log <subcommand> [parameters | global options]
2. aliyunlog configure <access_id> <access-key> <endpoint>
3. aliyunlog [--help | --version]
子命令与参数
日志服务命令行工具背后依赖于日志服务的Python SDK,
相关子命令对应于aliyun.log.LogClient
的方法,
参数和可选参数也一一对应. 具体支持的API参数, 请参考映射的Python SDK
API
例子:
def create_logstore(self, project_name, logstore_name, ttl=2, shard_count=30):
对应命令行:
> aliyunlog log create_logstore
--project_name=<value>
--logstore_name=<value>
[--ttl=<value>]
[--shard_count=<value>]
全局选项
所有命令都支持如下的全局选项:
[--access-id=<value>]
[--access-key=<value>]
[--region-endpoint=<value>]
[--client-name=<value>]
[--jmes-filter=<value>]
命令类别
项目组管理
项目组管理
- list_project
- create_project
- get_project
- delete_project
- copy_project
- 复制所有源project的logstore, logtail, machine group和index配置等到目标project中.
> aliyunlog log copy_project --from_project="p1" --to_project="p1" --to_client="account2"
- 注意:
to_client
是通过aliyunlog configure配置的其他账户, 也可以不传或传main
同域复制. - 参考跨域复制项目组配置了解如何跨域复制项目组配置.
日志库管理
日志库管理
- create_logstore
- delete_logstore
- get_logstore
- update_logstore
- list_logstore
分区管理
分区管理
- list_shards
- split_shard
- merge_shard
机器组管理
机器组管理
- create_machine_group
- 部分参数格式:
{
"machine_list": [
"machine1",
"machine2"
],
"machine_type": "userdefined",
"group_name": "group_name2",
"group_type": "Armory",
"group_attribute": {
"externalName": "ex name",
"groupTopic": "topic x"
}
}
- delete_machine_group
- update_machine_group
- get_machine_group
- list_machine_group
- list_machines
Logtail配置管理
Logtail配置管理
- create_logtail_config
- 参考创建Logtail配置了解如何创建各种格式的Logtail配置.
- update_logtail_config
- delete_logtail_config
- get_logtail_config
- list_logtail_config
机器组与Logtail配置关联
机器组与Logtail配置关联
- apply_config_to_machine_group
- remove_config_to_machine_group
- get_machine_group_applied_configs
- get_config_applied_machine_groups
索引管理
索引管理
- create_index
- 部分参数格式:
{
"keys": {
"f1": {
"caseSensitive": false,
"token": [
",",
" ",
"\"",
"\"",
";",
"=",
"(",
")",
"[",
"]",
"{",
"}",
"?",
"@",
"&",
"<",
">",
"/",
":",
"\n",
"\t"
],
"type": "text",
"doc_value": true
},
"f2": {
"doc_value": true,
"type": "long"
}
},
"storage": "pg",
"ttl": 2,
"index_mode": "v2",
"line": {
"caseSensitive": false,
"token": [
",",
" ",
"\"",
"\"",
";",
"=",
"(",
")",
"[",
"]",
"{",
"}",
"?",
"@",
"&",
"<",
">",
"/",
":",
"\n",
"\t"
]
}
}
- update_index
- delete_index
- get_index_config
- list_topics
游标操作
游标操作
- get_cursor
- get_cursor_time
- get_previous_cursor_time
- get_begin_cursor
- get_end_cursor
日志读写与消费
日志读写与消费
- put_logs
- 参数格式:
{
"project": "dlq-test-cli-35144",
"logstore": "logstore1",
"topic": "topic1",
"source": "source1",
"logtags": [
[
"tag1",
"v1"
],
[
"tag2",
"v2"
]
],
"hashKey": "1231231234",
"logitems": [
{
"timestamp": 1510579341,
"contents": [
[
"key1",
"v1"
],
[
"key2",
"v2"
]
]
},
{
"timestamp": 1510579341,
"contents": [
[
"key3",
"v3"
],
[
"key4",
"v4"
]
]
}
]
}
- get_logs
- 参数格式:
{
"topic": "",
"logstore": "logstore1",
"project": "dlq-test-cli-35144",
"toTime": "2018-01-01 11:11:11",
"offset": "0",
"query": "*",
"line": "10",
"fromTime": "2018-01-01 10:10:10",
"reverse": "true"
}
- 但参数
line
传入-1时, 就回获取所有. 但是当数据量很大, 超过1GB时, 最好使用get_log_all
- get_log_all
- 和
get_logs
一样, 但是迭代式获取数据并输出, 适合大容量的数据获取. - get_histograms
- pull_logs
- pull_log
- 和
pull_logs
类似, 但是迭代式获取数据并输出, 适合大容量的数据获取. - pull_log_dump
- 并发下载大量日志直接到本地文件,并一行一个日志。
投递管理
投递管理
- create_shipper
- 部分参数格式:
{
"oss_bucket": "dlq-oss-test1",
"oss_prefix": "sls",
"oss_role_arn": "acs:ram::1234:role/aliyunlogdefaultrole",
"buffer_interval": 300,
"buffer_mb": 128,
"compress_type": "snappy"
}
- update_shipper
- delete_shipper
- get_shipper_config
- list_shipper
- get_shipper_tasks
- retry_shipper_tasks
消费组管理
消费组管理
- create_consumer_group
- update_consumer_group
- delete_consumer_group
- list_consumer_group
- update_check_point
- get_check_point
Elasticsearch 数据迁移
Elasticsearch 数据迁移
- es_migrate
- 参考Elasticsearch 数据迁移了解如何将 Elasticsearch 中的数据导入日志服务。
其他配置管理
其他配置管理
- 报警管理
- 快速查询管理
- 报表管理
- 外部存储管理
参考命令行帮助
最佳实践
错误诊断
CLI默认把执行过程中的警告和错误存储在~/aliyunlogcli.log
中,
也可以配置~/.aliyunlogcli中__loggging__
来调整错误存储位置和格式:
[__logging__]
filename= # 默认是: ~/aliyunlogcli.log, 进行轮转
filebytes= # 默认是5242880 (5MB), 单个文件大小, 单位Byte
backupcount= # 默认是5个, 轮转文件的个数
#filemode= # 废弃
format= # 默认是: %(asctime)s %(threadName)s:%(levelname)s %(filename)s:%(lineno)d %(funcName)s %(message)s
datefmt= # 默认是: "%Y-%m-%d %H:%M:%S", 可以是strftime()兼容的日期/时间格式化字符串
level= # 默认是: warn, 可以是info, error, fatal, critical, debug
其他资源
- 日志服务产品介绍:http://www.aliyun.com/product/sls/
- 日志服务产品文档:https://help.aliyun.com/product/28958.html
- 日志服务Python SDK文档: http://aliyun-log-python-sdk.readthedocs.io/
- 其他问题请提工单
Best Practices¶
Configure CLI¶
Introduction¶
There’re ceratin options need to configure to make it more convenient to use CLI.
CLI options¶
CLI options include:
- Access Key and Endpoint
- About the definition of access key and endpoint,Refer to here
- Multiple account
- Configure multiple accounts for operation in specific region, cross region or testing purpose.
- Output format
- format the json output.
Configure AK and Endpoint¶
There’re three ways to configure the access key and endpoint and they’re prioritized as below:
- Parameters
> aliyunlog log create_project ..... --access-id=<value> --access-key=<value> --region-endpoint=<value>
Note: Any sub command support such way to overwrite the AK setings in later ways (env or config file) for the specific operations.
- Environment Variables
- ALIYUN_LOG_CLI_ACCESSID
- ALIYUN_LOG_CLI_ACCESSKEY
- ALIYUN_LOG_CLI_ENDPOINT
- Local configuration file
You could store them at ~/.aliyunlogcli
, the default section name is
main
[main]
access-id=
access-key=
region-endpoint=
Enable Https Connection¶
When configuring endpoint with prefix https://
, the connection
between CLI and Log service will be secured. Or else, it will use http
by default.
Modify the configuration file¶
Use the command “configure” to modify the configuration file:
> aliyunlog configure access_id access_key cn-beijing.log.aliyuncs.com
Multiple Account¶
- Store multiple accounts for some use cases (e.g. test, multiple region operations)
> aliyunlog configure access_id1 access_key1 cn-beijing.log.aliyuncs.com
> aliyunlog configure access_id2 access_key2 cn-hangzhou.log.aliyuncs.com test
AK is stored as:
[main]
access-id=access_id1
access-key=access_key1
region-endpoint=cn-beijing.log.aliyuncs.com
[test]
access-id=access_id2
access-key=access_key2
region-endpoint=cn-hangzhou.log.aliyuncs.com
Refer to here to know a simpler way to configure multiple accounts in config file.
- Use specific account
Any subcommand could use global opton --client-name=<value>
to use
specific configured account. e.g:
> aliyunlog log create_project ..... --client-name=test
It will use test
to create the project.
- Other Case
In some case, we need to operate cross regions, e.g.
> aliyunlog log copy_project --from_project="p1" --to_project="p1" --to_client=test
It will use account main
to copy project p1
in its region to
another region under account test
Default account¶
As mentions previously, the name of default account is main
, in
specific case, it’s necessary to swith to another account (in different
region). This could be archvied by using option --default-client
:
aliyunlog configure --default-client=beijing
This command will switch default account to the one beijing
.
Format output¶
JSON Format
Output is in json format normally, which is printed in one line in
console. In some cases, it’s useful to show them in pretty format. You
could use option --format-output=json
in each command for this:
aliyunlog log get_log.... --format-output=json
But if you want to make it a default behavior, configure it directly:
aliyunlog log configure --format-output=json
Not escape Non-ANSI
By default, for non-ANSI characters, it will escape it. If you want to
view the raw string, you could add a no_escape
to format of output:
aliyunlog log get_log.... --format-output=json,no_escape
It could be no_escape
or combine with other value in
format-output
with ,
And if you want to make it a default behavior, configure it directly:
aliyunlog log configure --format-output=json,no_escape
Get logs¶
Introduction¶
Get Log (GetLog) means searching out logs on certain conditions among specific time range basing on indexed data.
It has below characters:
- Just for index data.
- Against index and cross partition. Support get logs cross multiple logstores.
- Support search against topic, fulltext or fields etc.
- The time range is checking the logs’ time rather than the log rececieved time by server.
Refer to Query logs to learn how to turn on index and more syntax of searching.
Get log¶
Using CLI, it’s very easy to query logs basing on prepared search statements. Below is an example, it all slow access logs on a specific server among a specific time range.
aliyunlog log get_log_all --project="p1" --logstore="l1" --query="host:test.com and response_time>5000" --from_time="2018-01-24 16:00:00 CST" --to_time="2018-01-24 17:00:00 CST"
It get all logs that his the conditions
(host:test.com and response_time>5000
) from time range (from
2018-01-24 16:00:00 CST
to 2018-01-24 17:00:00 CST
) like:
[{"k1":"a1", "k2":"a2"}, {"k1":"b1", "k2":"b2"}, ... ]
Note: - We could also use another sub-command get_log(s)
passing
parameter size=-1
to get all logs hitting the condition. But it’s
recommended to use the sub-command get_log_all
when the data volume
is large (e.g. total logs count exceeds 1 million).
Transform the format and store to file¶
The example above shows the format of get log is a big JSON formatted
data, if we want to store the logs per row per log, we need to pass
parameter jmes-filter
to process and use redirection >>
to
output to disk.
Examples:
aliyunlog log get_log_all --project="p1" --logstore="l1" --query="host:test.com and response_time>5000" --from_time="2018-01-24 16:00:00 CST" --to_time="2018-01-24 17:00:00 CST" --jmes-filter="join('\n', map(&to_string(@), @))" >> ~/Desktop/test.data
It will store the results to file test.data
in below format:
{"k1":"a1", "k2":"a2"}
{"k1":"b1", "k2":"b2"}
...
Time format¶
The time format passed to the CLI is recommend as
%Y-%m-%d %H:%M:%S %Z
, like 2018-01-24 17:00:00 CST
. But the CLI
also support popular format like Jan 01 2018 10:10:10 CST
Note: The time-zone could not only be CST
but also any other
valid ones. Try to use more accurate time zone format like UTC-8
or
UTC+8
when you could not find the wanted time.
Actually, it’s also supported to use relative format like 2 day ago
,
now
, yesterday 0:0
, -1week
etc.
Time range¶
When passing time range parameters to get_log(s)
or get_log_all
,
please note:
- The time here points to log time parsed from raw logs when logs are uploaded (If not being specified, time received by server is used).
- The time range is closed for both left and right as
[]
, it means it will get logs at16:00:00
and17:00:00
in above example.
Query cross logstores¶
To get logs cross logstores, use sub-command get_project_logs
,
example:
aliyunlog log get_project_logs --request="{\"project\":\"p1\", \"query\":\"select count(1) from logstore1, logstore2, logstore3 where __date__ >'2017-11-10 00:00:00' and __date__ < '2017-11-13 00:00:00'\"}"
For more detail, please refer to document Query cross logstore.
最佳实践¶
配置CLI¶
介绍¶
和大多数CLI一样,CLI安装好后,需要适当配置一下才可以高效方便的使用。
有哪些CLI的配置项¶
CLI的配置项目包含如下内容:
- 服务入口和访问秘钥
- 关于服务入口和访问秘钥的定义,可以参考这里
- 多账户与默认账户
- 配置多个账户,以便访问特定区域,用于测试或者跨域操作等。
- 输出格式
- 修改输出json的格式化方式,默认是在一行打印。
- 修改输出非英文转义方式,默认转义.
配置秘钥与服务入口
日志服务CLI支持通过以下三种方式配置访问秘钥与入口,如果同时配置多种方式, 优先顺序是: 参数, 环境变量, 最后是本地配置文件.
- 参数
> aliyunlog log create_project ..... --access-id=<value> --access-key=<value> --region-endpoint=<value>
注意: 任意log子命令都支持以上方式定义特定的AK与Endpoint(覆盖后面的方式)
- 环境变量
- ALIYUN_LOG_CLI_ACCESSID
- ALIYUN_LOG_CLI_ACCESSKEY
- ALIYUN_LOG_CLI_ENDPOINT
- 本地配置文件
将存储AK与Endpoint在~/.aliyunlogcli, 默认使用的块名是main
[main]
access-id=
access-key=
region-endpoint=
使用HTTPS连接
当给服务入口地址加上https://
前缀时, CLI会使用https
连接,
否则默认使用http
.
修改配置文件
Configure命令可以修改配置文件内容.
> aliyunlog configure access_id access_key cn-beijing.log.aliyuncs.com
多账户
存储于多个账户¶
CLI支持以便在特定情况下使用(例如不同域操作、测试等):
> aliyunlog configure access_id1 access_key1 cn-beijing.log.aliyuncs.com
> aliyunlog configure access_id2 access_key2 cn-hangzhou.log.aliyuncs.com test
AK将存储为:
[main]
access-id=access_id1
access-key=access_key1
region-endpoint=cn-beijing.log.aliyuncs.com
[test]
access-id=access_id2
access-key=access_key2
region-endpoint=cn-hangzhou.log.aliyuncs.com
参考这里来了解一种简化的多账户配置方式。
使用特定账户¶
任意命令都可以通过选项--client-name=<value>
来使用特定配置的账户,
例如:
> aliyunlog log create_project ..... --client-name=test
将使用test
的AK来进行操作.
其他情况¶
某些情况下也需要跨账户操作, 例如:
> aliyunlog log copy_project --from_project="p1" --to_project="p1" --to_client=test
将main
账户下对应的项目p1
复制到账户test
下的p1
默认账户¶
如前面所属,默认账户是main
,在多用户情况下,切换默认账户也是很有用。通过选项--default-client
可以修改。
aliyunlog configure --default-client=beijing
这样其他命令默认会使用beijing
账户下的访问密码和服务入口地址。
输出格式
JSON格式
输出格式一般是以json形式输出,并且是打印在一行上面,某些情况下需要格式化输出方便查看,可以在特定命令上配置--format-output=json
,这样CLI会将输出进行格式化。
aliyunlog log get_log.... --format-output=json
如果期望所有输出都是这样,可以修改配置项来完成:
aliyunlog log configure --format-output=json
不转义非ANSI字符
对于非英文字符, 默认是转义的形式. 如果期望可以看到原始字符,
可以添加no_escape
到format-ouput
选项中:
aliyunlog log get_log.... --format-output=json,no_escape
与其他选项用逗号分隔即可. 如果期望所有输出都是这样,可以修改配置项来完成:
aliyunlog log configure --format-output=json,no_escape
使用CLI了解基础资源使用状况¶
背景¶
我们知道,日志服务的每个Project有一定的资源限制,你是否时常需要知道一些重要资源的使用状况?例如: 1. 一个Project的总shard数量是否已经超过了200个限制? 2. Logtail配置、机器组、报表数等是否快要满了?
现在使用CLI一行命令即可获得相关信息,以便及时提工单申请更大限额。
介绍¶
获取资源使用状况(get_resource_usage)检查传入的Project的所有基础资源,并以Json的形式列出,覆盖如下基础资源:
获取基础资源使用状况¶
一行命令即可获取:
例如:
aliyunlog log get_resource_usage --project="my project name" --format-output=json,no_escape
返回的格式是一个json,例如:
{
"consumer_group": {
"count": {
"status": 8
},
"logstores": {
"access-log": {
"limitation": 10,
"status": 7,
"usage": "70.0%"
},
"billdata": {
"limitation": 10,
"status": 1,
"usage": "10.0%"
}
}
},
"dashboard": {
"count": {
"limitation": 50,
"status": 28,
"usage": "56.0%"
}
},
"logstore": {
"count": {
"limitation": 200,
"status": 16,
"usage": "8.0%"
}
},
"logtail": {
"count": {
"limitation": 100,
"status": 2,
"usage": "2.0%"
}
},
"machine_group": {
"count": {
"limitation": 100,
"status": 1,
"usage": "1.0%"
}
},
"saved_search": {
"count": {
"limitation": 100,
"status": 11,
"usage": "11.0%"
}
},
"shard": {
"count": {
"limitation": 200,
"status": 30,
"usage": "15%"
},
"logstores": {
"access-log": {
"status": 2
},
"billdata": {
"status": 10
},
"test": {
"status": 2
},
"testmax": {
"status": 2
},
"tripdata": {
"status": 10
},
"vedio1": {
"status": 2
},
"web-1": {
"status": 2
}
}
}
}
注意: 1.
返回结构包含字段limitation
与usage
来表示限制与使用占比,但是这里的是默认的限制,如果通过工单增加过额度,这里并不会更新。
2.
这里传入了参数format-output
来格式化json,如果已经配置过整个CLI,可以省略。
单看shard消耗¶
有时候我们主要关心一个两个核心指标,并可能仅仅想要一个数字,可以通过jmes-filter
参数来进行处理,例如这里仅仅查看总的Shard使用情况,可以如下过滤:
aliyunlog log get_resource_usage --project="my project name" --jmes-filter="shard.count.status"
> 30
这里通过--jmes-filter="shard.count.status
直接获取输出结果中的shard的目前总数。
更多关于JMES过滤的信息,请参考这里。
使用CLI配置更多的shard¶
背景¶
我们知道,日志服务的每个Project有一定的资源限制,你是否遇到如下情况呢? 1. 创建Shard时,最多配置10个Shard,但是预期Logstore会有更大数据量写入尤其是查询读取需求? 2. 手工在Web控制台进行分裂,因为无法把握均匀分布,且点击较多,比较低效?
现在使用CLI一行命令即可配置Logstore到期望的数量了!
介绍¶
CLI通过命令arrange_shard
来自动将目标logstore的shard数量均匀分裂为期望的数量(最多到100个甚至更多)。
配置shard数量¶
例如:
aliyunlog log arrange_shard --project=my_project1 --logstore=my_logstore1 --count=50
这里配置目标logstore的shard数量为50个。
原理与注意事项¶
命令arrange_shard
根据期望shard的数量,以及目前shard分区范围,自动均匀的将目前分裂。
注意: 1.
这个命令也是通过split_shard
进行分裂,因此分裂过程中会制造出多个只读的shard。在一个Project最多有200个shard的情况下,如果一个Project只包含一个logstore,这个logstore可以配置为最多100个读写shard。
2.
因为服务器同步的原因,分裂命令完成后,在Web控制台一般需要1分钟左右可以看到最新shard数量。
在配置文件中简单配置多个账户¶
介绍¶
这里详细介绍一种通过配置文件,简单的配置多个账户的方式。
背景¶
在配置CLI里介绍了详细的CLI配置方法。如果要配置多个账户,可以通过CLI的configure
命令配置多个账户:
> aliyunlog configure access_id1 access_key1 cn-beijing.log.aliyuncs.com
> aliyunlog configure access_id2 access_key2 cn-hangzhou.log.aliyuncs.com test
AK将存储为:
[main]
access-id=access_id1
access-key=access_key1
region-endpoint=cn-beijing.log.aliyuncs.com
[test]
access-id=access_id2
access-key=access_key2
region-endpoint=cn-hangzhou.log.aliyuncs.com
如果我们有好多个域都需要配置,并且还存在多个权限的账户(如管理员和只读权限的秘钥),那么配置就有些复杂了。
方案¶
配置文件中,可以在DEFAULT
中集中配置多个权限的秘钥,再在其他区中用%(xxxx)s
的方式引用它即可,这样可以手动编辑~/.aliyunlogcli
来实现:
[DEFAULT]
admin_ak_id = id
admin_ak_key = key
user_ak_id = id
user_ak_key = key
[hz_admin]
access-id = %(admin_ak_id)s
access-key = %(admin_ak_key)s
region-endpoint = cn-hangzhou.sls.aliyuncs.com
[hz_user]
access-id = %(user_ak_id)s
access-key = %(user_ak_key)s
region-endpoint = cn-hangzhou.sls.aliyuncs.com
[bj_admin]
access-id = %(admin_ak_id)s
access-key = %(admin_ak_key)s
region-endpoint = cn-beijing.sls.aliyuncs.com
[bj_user]
access-id = %(user_ak_id)s
access-key = %(user_ak_key)s
region-endpoint = cn-beijing.sls.aliyuncs.com
创建Logtail配置¶
介绍¶
Logtail是日志服务提供的高性能低耗的日志收集客户端, 参考这里了解更多背景. 目前Logtail的配置有多种, 本节介绍如何快速有效的通过CLI创建各种Logtail的配置项.
有哪些Logtail的配置项¶
Logtail配置项目包含如下内容:
- 基于文本文件的:
- 极简模式
- JSON模式
- 分隔符模式
- 完整正则表达式模式
- syslog
- 自建产品
- NGNIX, 本节通过完整正则表达式来完成
- 飞天日志等
配置¶
1. 创建极简模式的logtail配置¶
极简模式是对文件进行单行分隔,
放入到名为content
的域中的最简单模式. 适用于简单存储处理的情况.
这里我们在项目组project1
中创建一个极简模式的logtail配置:
> aliyunlog log create_logtail_config --project_name="project1" --config_detail="file://simple_1.json"
文件simple_1.json
的内容如下:
{
"configName": "simple_1",
"inputDetail": {
"logType": "common_reg_log",
"logPath": "/user",
"filePattern": "*.log"
},
"inputType": "file",
"outputDetail": {
"logstoreName": "logstore1"
}
}
这里创建了一个属于日志库logstore1
的名叫simple_1
的logtail的配置.
主要的配置项就是日志的路径logPath
以及文件名模式filePattern
.
这里扫描/user
下包含深度子目录的所有.log
文件.
注意: 除了以上项目, 还需要如上所示填入相关信息:
其中inputType
是file
,
inputDetail.logType
是common_reg_log
.
时间配置¶
简单模式的日志时间采用的是系统收集的时间, 也就是抓取到这条日志的时间.
2. 创建JSON模式的logtail配置¶
JSON模式适用于文件本身就是JSON的情况.
这里我们在项目组project1
中创建一个JSON模式的logtail配置:
> aliyunlog log create_logtail_config --project_name="project1" --config_detail="file://json_1.json"
文件json_1.json
的内容如下:
{
"configName": "json_1",
"inputDetail": {
"logType": "json_log",
"filePattern": "*.json",
"logPath": "/json_1"
},
"inputType": "file",
"outputDetail": {
"logstoreName": "logstore1"
}
}
这里创建了一个属于日志库logstore1
的名叫json_1
的logtail的配置.
主要的配置项就是日志的路径logPath
以及文件名模式filePattern
.
这里扫描/user
下包含深度子目录的所有.json
文件.
注意:
- 这里需要按照如上的模板填入相关信息.
其中
inputType
必须是file
,inputDetail.logType
是json_log
. - Logtail会根据json的格式自动提取对应的域信息. 例如一个旅行日志的内容如下:
{
"from": "nanjing",
"to": "shanghai",
"people": "xiaoming",
"travel_time": "2018-1-1 10:10:10"
}
会自动配置域from
, to
和people
和travel_time
.
时间配置¶
默认情况下, 日志的时间会使用系统时间, 也就是抓取到这条日志的时间.
如果日志本身包含了更加准确的时间, 例如上面旅行日志的例子,
可以在logtail的配置中指定travel_time
为日志时间如下:
{
"inputDetail": {
"timeFormat": "%Y-%M-%D %h:%m:s",
"timeKey": "travel_time"
}
}
这里通过inputDetail.timeKey
指定travel_time
为时间关键字,
通过inputDetail.timeFormat
配置其格式为%Y-%M-%D %h:%m:s
,
针对时间格式,
可以进一步参考这里
3. 创建分隔符模式的logtail配置¶
分隔符模式适用于文件本身基于某个分隔符分隔的文件, 例如CSV, TSV等等.
日志服务支持最多3个字符组合的分隔, 例如一个旅行日志的文件内如是:
2017-1-1 10:10:00&#&nanjing&#&shanghai&#&xiao ming
2017-1-1 20:10:00&#&beijing&#&hangzhou&#&xiao wang
分隔符就是&#&
可以分隔出时间
, 出发城市
, 达到城市
,
旅客
.
这里我们在项目组project1
中创建一个分隔符模式的logtail配置:
> aliyunlog log create_logtail_config --project_name="project1" --config_detail="file://sep_1.json"
文件sep_1.json
的内容如下:
{
"configName": "sep_1",
"logSample": "2017-1-1 10:10:00&#&nanjing&#&shanghai&#&xiao ming",
"inputDetail": {
"logType": "delimiter_log",
"logPath": "/user",
"filePattern": "travel.log",
"separator": "&#&",
"key": [
"travel_time",
"from_city",
"to_city",
"people"
]
},
"inputType": "file",
"outputDetail": {
"logstoreName": "logstore1"
}
}
这里创建了一个属于日志库logstore1
的名叫json_1
的logtail的配置.
主要的配置包括:
- 日志的路径以及文件名模式.
这里扫描
/user
下包含深度子目录的所有travel.json
文件. - 分隔符为
&#&
, 以及对应的域名, 这里是travel_time
,from_city
,to_city
和people
. - 值得注意的是, 也需要在域
logSample
中填写一个例子. - 其他需要按照如上的模板填入相关信息.
其中
inputType
必须是file
,inputDetail.logType
是delimiter_log
.
注意, 多个域的分隔符必须是一致的, 例如下面的文件,期望配置分隔符&|
,
并不会起作用.
2017-1-1 10:10:00&nanjing|&shanghai|xiao ming
2017-1-1 20:10:00|beijing&hangzhou|xiao wang
因为这里的文件分隔符并不是一致的, 这种情况就需要用到完整正则表达式的方式了.
4. 创建不可见字符的分隔的logtail配置¶
对于一些不可见字符, 可以在separator
域中使用转义:
特殊字符 | 字符 |
---|---|
双引号 | " |
反斜杠 | \ \ |
斜杠 | \/ |
回退符 | \b |
换页符 | \f |
换行符 | \n |
回车符 | \r |
横向制表符 | \t |
纵向制表符 | \v |
\u 0000 | 任意字符 |
例如,这里我们在项目组project1
中创建一个分隔符为\u0001
的的logtail配置:
> aliyunlog log create_logtail_config --project_name="project1" --config_detail="file://sep_2.json"
文件sep_2.json
的内容如下:
{
"configName": "sep_1",
"logSample": "2017-1-1 10:10:00\u0001nanjing\u0001shanghai\u0001xiao ming",
"inputDetail": {
"logType": "delimiter_log",
"logPath": "/user",
"filePattern": "travel.log",
"separator": "\u0001",
"key": [
"travel_time",
"from_city",
"to_city",
"people"
]
},
"inputType": "file",
"outputDetail": {
"logstoreName": "logstore1"
}
}
注意:
通过CLI配置了不可见字符作为分隔符后,再进入Web控制台编辑这个Logtail配置页面是看不到这些字符,但点击按钮下一步
继续其他的配置是不受影响的。
5. 创建正则表达式模式的logtail配置¶
正则表达式模式利用了强大的正则表达式的解析能力, 提取非结构化文本中的特定模式的域. 适用于文件本身结构较为特殊的情况. 完整的正则表达式可以参考这里
一个典型的例子是NGNIX日志:
10.1.1.1 - - [2018-1-1 10:10:10] "GET / HTTP/1.1" 0.011 180 404 570 "-" "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; 360se)"
10.1.1.1 - - [2018-1-1 10:10:20] "GET / HTTP/1.1" 0.011 180 404 570 "-" "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; 360se)"
可以看到模式是:
IP - - [时间] "HTTP请求" 响应时间 请求大小 HTTP状态号 响应大小 "-" "客户端"
这里用对应的正则表达式是:
(\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3}) - - \[([^\]+])\] "([\w\s\/\.]+) ([\d\.]+) ([\d\.]+) ([\d\.]+) ([\d\.]+) "-" "[^\"]+"
这里我们在项目组project1
中创建一个正则表达式模式的logtail配置:
> aliyunlog log create_logtail_config --project_name="project1" --config_detail="file://reg_1.json"
文件reg_1.json
的内容如下:
{
"configName": "ngnix_1",
"logSample": "10.1.1.1 - - [13/Mar/2016:10:00:10 +0800] \"GET / HTTP/1.1\" 0.011 180 404 570 \"-\" \"Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; 360se)\"",
"inputDetail": {
"logType": "common_reg_log",
"logPath": "/ngnix",
"filePattern": "*.log",
"regex": "(\\d{1,3}\\.\\d{1,3}\\.\\d{1,3}\\.\\d{1,3}) - - \\[([^\\]+])\\] \"([\\w\\s\/\\.]+) ([\\d\\.]+) ([\\d\\.]+) ([\\d\\.]+) ([\\d\\.]+) \"-\" \"[^\\\"]+\"",
"key": [
"client_ip",
"request_time",
"method_type",
"process_period",
"request_bytes",
"http_status",
"response_bytes",
"client_agent"
],
"timeFormat": "%Y-%M-%D %h:%m:%s"
},
"inputType": "file",
"outputDetail": {
"logstoreName": "logstore1"
}
}
这里创建了一个属于日志库logstore1
的名叫ngnix_1
的logtail的配置.
主要的配置包括:
- 日志的路径以及文件名模式.
这里扫描
/ngnix
下包含深度子目录的所有*.log
文件. - 正则表达式写在
regex
中, 以及对应的域名放在key
中. - 这里也需要在域
logSample
中填写一个例子. - 其他需要按照如上的模板填入相关信息.
其中
inputType
必须是file
,inputDetail.logType
是delimiter_log
.
时间配置¶
默认情况下, 日志的时间会使用系统时间, 也就是抓取到这条日志的时间.
如果日志本身包含了更加准确的时间, 例如上面NGNIX日志的例子,
可以在logtail的配置中指定request_time
为日志时间如下:
{
"inputDetail": {
"timeKey": "request_time",
"timeFormat": "%Y-%M-%D %h:%m:s"
}
}
这里通过inputDetail.timeFormat
配置其格式为%Y-%M-%D %h:%m:s
,
针对时间格式,
可以进一步参考这里
注意:
如果只配置了inputDetail.timeFormat
而没有配置inputDetail.timeKey
,
默认使用域time
.
6. 创建syslog的logtail配置¶
syslog与文件方式互补, 收集部署上更方便. 除了通过CLI进行配置外, 还需要在logtail客户端通过本地配置文件的方式进行进一步配置. 具体参考这里.
这里我们在项目组project1
中创建一个syslog模式的logtail配置:
> aliyunlog log create_logtail_config --project_name="project1" --config_detail="file://syslog_1.json"
文件syslog_1.json
的内容如下:
{
"configName": "syslog_1",
"inputDetail": {
"tag": "tag1"
},
"inputType": "syslog",
"outputDetail": {
"logstoreName": "logstore1"
}
}
这里创建了一个属于日志库logstore1
的名叫syslog_1
的logtail的配置.
主要的配置包括:
- 标签
tag
. - 其他需要按照如上的模板填入相关信息.
其中
inputType
必须是syslog
, 注意这里没有域inputDetail.logType
.
高级配置项¶
配置多行日志¶
很多应用的日志可能偶尔会出现多行的情况,
例如Java的日志会打印一些堆栈信息. 这种情况就需要多行配置. 注意:
目前仅正则表达式模式
支持多行配置.
例如:
com.journaldev.log.LoggingExample::main::2017-1-1 01:42:43::Msg977
com.journaldev.log.LoggingExample::main::2017-1-1 03:42:43::Break at
File "/Applications/PyCharm CE.app/Contents/helpers/pydev/pydevd.py", line 1599, in <module>
globals = debugger.run(setup['file'], None, None, is_module)
File "/Applications/PyCharm CE.app/Contents/helpers/pydev/pydevd.py", line 1026, in run
pydev_imports.execfile(file, globals, locals) # execute the script
File "/Users/wjo1212/GithubProjects/aliyun-log-cli/aliyunlogcli/cli.py", line 5, in <module>
main()
com.journaldev.log.LoggingExample::main::2017-1-1 05:42:43::Msg978
这种情况就需要配置logBeginRegex
来指定每一条日志的首行模式:
{
"inputDetail": {
"logBeginRegex": "com.journaldev.log.LoggingExample::main::.*"
}
}
这里配置了inputDetail.logBeginRegex
指定了每行新的日志都是以com.journaldev.log.LoggingExample::main::.*
作为第一行的.
注意:
这里的正则表达式最后使用了.*
对首行进行完整匹配.
这一点是必须的, 例如如下的配置将不起作用:
{
"inputDetail": {
"logBeginRegex": "com.journaldev.log.LoggingExample::main::"
}
}
Logtail客户端的解析配置¶
Logtail也支持针对特定目标文件, 定义一些高级行为, 所有基于文本的收集方式都支持如下配置:
- 本地缓存 (localStorage):
- 当日志服务不可用时,日志缓存在机器本地目录,服务恢复后进行续传,默认最大缓存值1GB
- 默认打开(true)
- 上传原始日志 (enableRawLog):
- 开启该功能后默认会新增字段将原始日志内容一并上传
- 默认关闭 (false)
- Topic生成方式 (topicFormat):
- 支持:
- 空(
none
): 不配置主题. - 机器组(
group_topic
): 基于所引用的机器组的Topic属性来配置. - 基于文件名的正则表达式: 用正则式从路径里提取一部分内容作为Topic。可以用于区分具体用户或实例产生的日志数据。
- 空(
- 关于主题, 查考这里
- 默认是空-不生成主题(none)
- 支持:
- 日志文件编码 (fileEncoding):
- 目前支持
utf8
和gbk
- 默认是
utf8
- 目前支持
- 最大监控目录深度 (maxDepth):
- 指定从日志源采集日志时,监控目录的最大深度,即最多监控几层日志。最大目录监控深度范围0-1000,0代表只监控本层目录。
- 默认是100
- 超时属性 (preserve)
- 如果一个日志文件在指定时间内没有任何更新,则认为该文件已超时。您可以对超时属性指定以下配置。
永不超时
(true): 指定持续监控所有日志文件,永不超时。超时30分钟超时
(false): 如日志文件在30分钟内没有更新,则认为已超时,并不再监控该文件。
- 默认
永不超时
(true)
- 如果一个日志文件在指定时间内没有任何更新,则认为该文件已超时。您可以对超时属性指定以下配置。
- 最大超时目录深度 (preserveDepth):
- 当配置为
30分钟超时
时, 需要配置深度, 范围1-3
- 当配置为
- 过滤器配置:
- 日志只有完全符合过滤器中的条件才会被收集。
- 分别配置过滤关键字
filterKey
和对应内容正则表达式filterRegex
.
如下示例, 这里关闭了本地存储
, 打开了原始文件上传
,
配置了一个自定义的基于正则表达式的Topic方式
, 文件编码为gbk
,
最大目录深度200
, 检测文件变化
30分钟, 对应深度为3.
同时配置了只抓取关键字from_city
和to_city
满足特定正则的日志..
{
"inputDetail": {
"localStorage": false,
"enableRawLog": true,
"topicFormat": "/user.+/(\\w+).log",
"fileEncoding": "gbk",
"maxDepth": 200,
"preserve": false,
"preserveDepth": 3,
"filterKey": [
"from_city",
"to_city"
],
"filterRegex": [
"nanjing|shanghai",
"beijing|hangzhou"
]
}
}
后续操作¶
完成Logtail的配置后, 还需要应用配置到机器组, 可以通过操作apply_config_to_machine_group来完成.
跨域复制项目配置¶
介绍¶
日志服务是在域(Region)范围内以项目(Project)为边界进行操作的, 典型场景下客户会在多个域部署同一套日志项目和日志库来接受日志和查询日志的服务. 因此常常需要跨域复制多个项目的配置. 使用日志服务的CLI可以比较容易的达到这点.
哪些数据需要复制¶
通常一个日志服务的配置包含以下部分:
- 多个日志库(logstore), 以及其下面的
- 索引配置
- logtail配置
日志库也包含以下配置, 但是一般多个区域内的配置会有所不同, 一般不需要复制.
- 机器组
- 机器组和logtail配置的关联
注意: CLI也支持机器组相关复制, 但需要额外参数打开.
复制项目配置到其他域¶
这里假设已经完成了CLI的安装.
1. 配置多区域账户¶
首先CLI中配置多个区域账户, 以便后续操作, 这一步也是一次性的. 如果之前已经做过, 这里可以跳过.
这里配置2个域的账户, 一个杭州公有云, 一个北京公有云.
> aliyunlog configure AKID****123 AKKEY****123 cn-hangzhou.log.aliyuncs.com hz
> aliyunlog configure AKID****123 AKKEY****123 cn-beijing.log.aliyuncs.com bj
注意: 这里的最后一个参数, 用于表明这个账户的名字, 以便后续使用.
不传入默认是main
, 也是默认使用的账户.
参考:
2. 复制项目组配置到目标域¶
假设我们需要将杭州公有云的项目组project1
复制到北京公有云,
这可以这样操作:
> aliyunlog log copy_project --from_project="project1" --to_project="project1" --client-name=hz --to_client=bj
这里CLI就会使用账户hz
来读取源项目组project1
的相关配置,
并使用账户bj
来构建一个新的项目组project1
.
如果使用默认账户读取读取源项目组, 可以省略参数--client-name
3. 同域项目组配置复制¶
某些特殊情况, 我们需要在同一个域内复制项目组并保留所有配置项.
这里我们复制账户bj
下的project1
为project2
:
> aliyunlog log copy_project --from_project="project1" --to_project="project2" --copy_machine_group=true --client-name=bj
提示:
传入copy_machine_group
为true
可以复制机器组的配置以及Logtail配置的管理信息.
拉取日志¶
介绍¶
拉取数据(PullLog)指的是针对特定分区, 按照游标来消费日志, 游标指的是服务器接收到日志的时间. 拉取数据的特点是:
- 没有索引的数据也可以拉取.
- 按照分区粒度拉取, 每个分区游标互相无关.
- 速度快, 易并发, 且支持传输压缩.
- 拉取的数据按照服务器接受的时间排序.
并发下载¶
命令pull_log_dump
针对下载做了优化,直接指定文件路径,即可将日志下载在本地。
aliyunlog log pull_log_dump --project_name="p1" --logstore_name="l1" --from_time="2018-01-24 16:00:00 CST" --to_time="2018-01-24 17:00:00 CST" --file_path="/data/dump_{}.data"
这里拉取从时间2018-01-24 16:00:00 CST
到2018-01-24 17:00:00 CST
并把数据下载本地,因为是并发,所以会存储在多个文件中,文件名根据/data/dump_{}.data
中的{}
替换成多个并发的数字(一般是分区号
)。文件的内容格式为一行一个日志,例如:
{"k1":"a1", "k2":"a2"}
{"k1":"b1", "k2":"b2"}
...
下载完后会显示下载的文件列表以及日志数,例如:
{"files": {"/data/dump_1.data": 75, "/data/dump_2.data": 14}, "total_count": 89}
时间格式¶
时间格式推荐是%Y-%m-%d %H:%M:%S %Z
,
如2018-01-24 17:00:00 CST
, 但也支持其他合法的时间格式,
例如:Jan 01 2018 10:10:10 CST
注意: 时区不仅可以是CST,如果发现拿不到数据,可以改成特定时区例如 UTC-8 或者 UTC+8
时间范围¶
命令pull_log_dump
传入的时间范围, 需要注意几点:
- 这里的时间指的是服务器接受日志的时间
- 时间的范围是左闭右开
[)
, 上面例子中16:00:00
服务器接受到的日志会被拉取到, 但是17:00:00
服务器所接受到的日志不会.
特殊的时间范围¶
有时我们不关心数据的某一边界, 例如期望获得所有存储的数据到某一天日期位置. 或者获取从某一天开始到目前存储的数据为止的话, 就可以使用特殊游标.
特殊游标包括begin
和end
. 例如:
aliyunlog log pull_log_dump --project_name="p1" --logstore_name="l1" --shard_id=3 --from_time="begin" --to_time="2018-01-24 17:00:00 CST"
这里拉取所有2018-01-24 17:00:00 CST
之前服务器接收到的日志.
又例如:
aliyunlog log pull_log_dump --project_name="p1" --logstore_name="l1" --shard_id=3 --from_time="2018-01-24 17:00:00 CST" --to_time="end"
这里拉取所有2018-01-24 17:00:00 CST
开始及之后服务器接收到的日志.
查询日志¶
介绍¶
查询数据(GetLog)指的是针对索引数据, 在特定时间范围内搜索特定的日志.
查询日志的特点是:
- 针对获取索引的数据
- 按照索引获取, 跨分区, 支持跨多个logstore查询
- 可以按照主题, 全文分词, 特定索引域等查询条件获取.
- 按照日志自身时间查询, 而不是服务器接收.
参考查询语法以了解如何打开索引以及查询的相关语法.
查询日志¶
根据设定好的查询语句查询数据, 即可通过CLI轻松查询数据. 下面例子查询某个时间范围, 某台服务器响应速度大于5秒的访问日志.
aliyunlog log get_log_all --project="p1" --logstore="l1" --query="host:test.com and response_time>5000" --from_time="2018-01-24 16:00:00 CST" --to_time="2018-01-24 17:00:00 CST"
这里拉取从时间2018-01-24 16:00:00 CST
到2018-01-24 17:00:00 CST
在内满足条件host:test.com and response_time>5000
的日志,
例如:
[{"k1":"a1", "k2":"a2"}, {"k1":"b1", "k2":"b2"}, ... ]
注意:
- 这里也可以使用子命令
get_log(s)
并传入size=-1
, 但如果数据量特别多时, 例如总行数100万行的时候, 推荐使用get_log_all
.
转换格式并存储到文件¶
如果期望将数据按照一行一条的形式存储下来,
一般需要加入jmes-filter
进行处理. 如果期望存储到文件,
这直接使用>>
重定向到文件即可.
例如:
aliyunlog log get_log_all --project="p1" --logstore="l1" --query="host:test.com and response_time>5000" --from_time="2018-01-24 16:00:00 CST" --to_time="2018-01-24 17:00:00 CST" --jmes-filter="join('\n', map(&to_string(@), @))" >> ~/Desktop/test.data
输出将被存储在文件test.data
中, 格式为:
{"k1":"a1", "k2":"a2"}
{"k1":"b1", "k2":"b2"}
...
时间格式¶
时间格式推荐是%Y-%m-%d %H:%M:%S %Z
,
如2018-01-24 17:00:00 CST
, 但也支持其他合法的时间格式,
例如:Jan 01 2018 10:10:10 CST
注意: 时区不仅可以是CST,如果发现拿不到数据,可以改成特定时区例如 UTC-8 或者 UTC+8
CLI还支持更多其他格式的时间格式,例如2 day ago
等,参考这里.
时间范围¶
命令get_log(s)
或get_log_all
传入的时间范围, 需要注意几点:
- 这里的时间指的是解析出的日志时间(日志配置没有指定的情况下, 服务器接收日志的时间会设为这个时间).
- 时间的范围是左闭右闭
[]
, 上面例子中16:00:00
和17:00:00
时间的日志都会获得.
使用CLI进行高速跨域日志复制、历史数据重新索引与数仓投递¶
背景¶
使用日志服务是不是常常遇到如下烦恼? 1. 开启了字段索引却无法对历史日志起作用,而手动重建索引又很困难怎么办? 2. 需要迁移数据,复制数据到其他区域logstore,写代码实现大并发复制又很复杂怎么办? 3. 投递日志到OSS/ODPS仅仅对新数据起作用,又想投递历史日志怎么办?
现在使用CLI就可以帮你轻松实现这些操作。
介绍¶
复制数据(CopyData)支持将特定时间范围内的logstore的数据复制到特定logstore中去。其具备如下一些特点:
- 没有索引的数据也可以同步.
- 速度快, 易并发, 且支持传输压缩.
- 拉取的数据按照服务器接受的时间排序.
- 支持跨域、跨项目库复制。
- 支持复制数据到同一个logstore(重新索引)。
前提¶
这里假设已经完成了CLI的安装.
配置多区域账户¶
首先CLI中配置多个区域账户, 以便后续操作, 这一步也是一次性的. 如果之前已经做过, 这里可以跳过.
这里配置2个域的账户, 一个杭州公有云, 一个北京公有云.
> aliyunlog configure AKID****123 AKKEY****123 cn-hangzhou.log.aliyuncs.com
> aliyunlog configure AKID****123 AKKEY****123 cn-beijing.log.aliyuncs.com bj
注意: 这里的最后一个参数, 仅仅用于表明这个账户的名字, 以便后续使用.
不传入默认是main
, 也是默认使用的账户,
这里使用杭州的账户作为默认账户.
参考:
重新索引¶
如果因为某些特定原因,某个时间范围内的日志没有建立索引,无法被查询和统计。可以如下操作将日志重新写入,就可以实现重新索引的效果。
例如:
aliyunlog log copy_data --project="源project" --logstore="源logstore" --from_time="2018-09-05 0:0:0 CST" --to_time="2018-09-06 0:0:0 CST"
这里将杭州区域的源project
的源logstore
中服务器在时间范围[“2018-09-05
0:0:0 CST”,”2018-09-06 0:0:0
CST”)内接收到的数据,重新写入到源logstore
中去。
注意: 这里仅仅是复制一份数据进入目标logstore,并在写入时自动对其索引,原来的没有被索引的日志依然存在。
跨区域复制数据¶
有时需要将某一个logstore的日志迁移到另外一个logstore中去时,可以如下操作:
准备好目标logstore¶
假设目标logstore已经创建好了,并且配置好了索引。这一步操作可以在Web控制台完成,也可以通过CLI的复制logstore配置来完成,或者使用CLI的create_logstore创建日志库,再配置索引,通过命令get_index_config获取索引,调用命令create_index来实现。
复制数据¶
例如:
aliyunlog log copy_data --project="源项目" --logstore="源logstore" --from_time="2018-09-05 0:0:0 CST" --to_time="2018-09-06 0:0:0 CST" --to_project="目标project" --to_logstore="目标logstore" --to_client="bj"
这里将杭州区域的源project
的源logstore
中服务器在时间范围[“2018-09-05
0:0:0 CST”,”2018-09-06 0:0:0
CST”)内接收到的数据,写入到北京
区域的目标project
的目标logstore
中去。
注意:
这里用--to_client
指定操作目标project的账户为bj
,是前面前提中配置的账户名。
投递历史日志到OSS/ODPS¶
日志服务的投递任务配置好后,仅仅对新接受数据产生作用。这里也可以借助复制数据来实现投递历史日志的效果:
- 创建一个临时的logstore(不需要配置索引)
- 在临时logstore上配置投递OSS/ODPS的任务
- 复制需要投递的日志到目标logstore
- 投递结束后,删除临时logstore
时间格式¶
时间格式推荐是%Y-%m-%d %H:%M:%S %Z
,
如2018-01-24 17:00:00 CST
, 但也支持其他合法的时间格式,
例如:Jan 01 2018 10:10:10 CST
注意: 时区不仅可以是CST,如果发现拿不到数据,可以改成特定时区例如 UTC-8 或者 UTC+8
CLI还支持更多其他格式的时间格式,例如2 day ago
等,参考这里.
时间范围¶
传入的时间范围, 需要注意几点:
- 这里的时间指的是服务器接受日志的时间
- 时间的范围是左闭右开
[)
, 上面例子中16:00:00
服务器接受到的日志会被拉取到, 但是17:00:00
服务器所接受到的日志不会.
特殊的时间范围¶
有时我们不关心数据的某一边界, 例如期望获得所有存储的数据到某一天日期位置. 或者获取从某一天开始到目前存储的数据为止的话, 就可以使用特殊游标.
特殊游标包括begin
和end
. 例如:
aliyunlog log copy_data --project="p1" --logstore="l1" --from_time="begin" --to_time="2018-01-24 17:00:00 CST" --to_logstore="l2"
这里复制所有2018-01-24 17:00:00 CST
之前服务器接收到的日志到logstorel2
。
又例如:
aliyunlog log copy_data --project="p1" --logstore="l1" --from_time="2018-01-24 17:00:00 CST" --to_time="end" --to_logstore="l2"
这里复制所有2018-01-24 17:00:00 CST
开始及之后服务器接收到的日志到logstorel2
。
使用JMES表达式过滤处理日志¶
内容敬请期待
日志上传¶
内容敬请期待
使用CLI的灵活时间格式来执行查询、报警和下载等¶
介绍¶
在查询、统计、拉取时需要传入时间范围,但是经常我们需要传入更灵活的参数,如:”今天凌晨”,”昨天”,”3天前”,”现在”等参数时,大脑需要费力的换算成对应的时间格式,如”2018-8-1 10:10:00”等。 另一方面,用脚本定期跑的时候,需要相对这样的相对值,例如,每5分钟查询,过去60分钟的范围的数据是否有异常等。每次执行希望有一个时间范围是:过去60分钟到现在。
现在这样的方式CLI支持了!
支持的方法¶
以下方法的参数,如from_time
、to_time
等目前也接受相对时间了。
方法 | 作用 |
---|---|
get_log | 查询统计数据(参数放在文件中),未完成时自动重试。 |
get_logs | 查询统计数据(参数放在命令行中),未完成时自动重试。 |
get_log_all | 迭代式查询更大量的数据,未完成时自动重试。 |
pull_log | 基于服务器接受时间范围,批量拉取特定shard的数据。 |
pull_log_dump | 并发批量下载logstore所有shard的数据到本地。 |
get_cursor | 获取特定服务器时间特定shard的游标(以便基于游标拉取数据) |
get_histograms | 获取查询范围内的日志分布。 |
支持的时间格式¶
除了常规的格式,例如:”%Y-%m-%d %H:%M:%S CST”的例子”2018-01-02 12:12:10 CST”; 也支持”1 hour ago”, “now”, “yesterday 0:0:0”等格式了。更多格式也可以参考https://dateparser.readthedocs.io/en/latest/
相对1分钟: -1min ~ now, 或 1 min ago ~ now
相对4小时: -4h ~ now,或 4 hours ago ~ now 或 4 hour ago
相对4天(24小时): -1d ~ now,或 1 day ago ~ now
相对1周(7x24小时): -1week ~ now, 或 1 week ago ~ now
今天: today 0:0:0 ~ now
昨天: yesterday 0:0:0 ~ yesterday 23:59:59,或 1 day ago 0:0:0 ~ -1d 23:59:59
前天: the day before yesterday 0:0:0 ~ 2 day ago 23:59:59
例子¶
实时显示当前网站UV¶
如下命令
aliyunlog log get_log --project=pdata1 --logstore=log1 --from_time="today 00:00:00" --to_time="now" --query="host: www.dcd.mock-domain.com | select count(distinct remote_addr) as uv" --jmes-filter="join('\n', map(&to_string(uv), @))"
每次运行时都会显示今天的独立访问客户的数量
定期导出客户端IP列表¶
aliyunlog log get_log --project=pdata1 --logstore=log1 --from_time="yesterday 00:00:00" --to_time="yesterday 23:59:59" --query="host: www.dcd.mock-domain.com | select distinct remote_addr" --jmes-filter="join('\n', map(&to_string(remote_addr), @))"
以上命令可以在每天0:0:1运行,导出昨天的独立访问客户的列表
Elasticsearch 数据迁移¶
概述¶
使用日志服务 CLI 可以方便您快速将 Elasticsearch 中的数据导入日志服务。
配置¶
aliyun-log-cli.readthedocs.io 无法正常显示表格,请参阅tutorial_es_migration_cn.md
数据映射¶
logstore - index¶
CLI 默认会将 Elasticsearch index 中的数据迁移至同名的 logstore 中,当然您也可以通过参数 logstore_index_mappings 指定将多个 index 中的数据迁移至一个 logstore。
logstore 不必事先创建,如果 CLI 发现目标 logstore 未创建,会为您在指定的 project 下创建好。
数据类型映射¶
CLI 会根据 Elasticsearch 的数据类型 在index 对应的 logstore 中创建好索引。
- Core datatypes
Elasticsearch | 日志服务 |
---|---|
text | text |
keyword | text,不分词 |
long | long |
integer | long |
short | long |
byte | long |
double | double |
float | double |
half_float | double |
scaled_float | double |
date | text |
boolean | text,不分词 |
binary | n/a |
integer_range | json |
float_range | json |
long_range | json |
double_range | json |
date_range | json |
ip_range | text,不分词 |
- Complex datatypes
Elasticsearch | 日志服务 |
---|---|
Array datatype | n/a |
Object datatype | json |
Nested datatype | n/a |
- Geo datatypes
Elasticsearch | 日志服务 |
---|---|
Geo-point datatype | text |
Geo-Shape datatype | text |
- Specialised datatypes
Elasticsearch | 日志服务 |
---|---|
IP datatype | text,不分词 |
Completion datatype | n/a |
Token count datatype | n/a |
mapper-murmur3 | n/a |
Percolator type | n/a |
join datatype | n/a |
抓取模式¶
- 为了提高吞吐量,CLI 会为每个 index 的每个 shard 创建一个数据迁移任务,并提交到内部进程池中执行。
- 当全部任务执行完成后,CLI 才会退出。
任务执行情况展示¶
CLI 默认把任务执行情况存储在~/aliyunlogcli.log中, 也可以配置~/.aliyunlogcli中__loggging__来调整信息存储位置和格式:
[__logging__]
filename=~/es_migration.log
level=info
- 单个迁移任务执行结果展示。
========Tasks Info========
...
task_id=1, slice_id=1, slice_max=10, hosts=localhost:9200, indexes=None, query=None, project=test-project, time_cost_in_seconds=128.71100688, status=CollectionTaskStatus.SUCCESS, count=129330, message=None
...
编号为 1 的迁移任务执行成功,耗时 128.7s,迁移文档数量 129330。
- 迁移任务执行结果汇总信息。
========Summary========
Total started task count: 10
Successful task count: 10
Failed task count: 0
Total collected documentation count: 1000000
MigrationManager 总共启动了 10 个数据数据迁移任务,全部执行成功。迁移文档总数 1000000。
使用样例¶
- 将 hosts 为
localhost:9200
的 Elasticsearch 中的所有文档导入日志服务的项目project1
中。
aliyunlog log es_migration --hosts=localhost:9200 --project_name=project1
- 指定将 Elasticsearch 中索引名以
myindex_
开头的数据写入日志库logstore1
,将索引index1,index2
中的数据写入日志库logstore2
中。
aliyunlog log es_migration --hosts=localhost:9200,other_host:9200 --project_name=project1 --logstore_index_mappings='{"logstore1": "myindex_*", "logstore2": "index1,index2"}}'
- 使用参数 query 指定从 Elasticsearch 中抓取
title
字段等于python
的文档,并使用文档中的字段date1
作为日志的 time 字段。
aliyunlog log es_migration --hosts=localhost:9200 --project_name=project1 --query='{"query": {"match": {"title": "python"}}}'
使用 HTTP 基本认证
user:secret@localhost:9200
,从 Elasticserch 中迁移数据。aliyunlog log es_migration --hosts=user:secret@localhost:9200 --project_name=project1
Mapped SDK API¶
Request and Config Class¶
GetHistogramsRequest ([project, logstore, …]) |
The request used to get histograms of a query from log. |
GetLogsRequest ([project, logstore, …]) |
The request used to get logs by a query from log. |
GetProjectLogsRequest ([project, query]) |
The request used to get logs by a query from log cross multiple logstores. |
ListTopicsRequest ([project, logstore, …]) |
The request used to get topics of a query from log. |
ListLogstoresRequest ([project]) |
The request used to list log store from log. |
PutLogsRequest ([project, logstore, topic, …]) |
The request used to send data to log. |
LogtailConfigGenerator |
Generator of Logtial config |
PluginConfigDetail (logstoreName, configName, …) |
The logtail config for simple mode |
SeperatorFileConfigDetail (logstoreName, …) |
The logtail config for separator mode |
SimpleFileConfigDetail (logstoreName, …[, …]) |
The logtail config for simple mode |
FullRegFileConfigDetail (logstoreName, …[, …]) |
The logtail config for full regex mode |
JsonFileConfigDetail (logstoreName, …[, …]) |
The logtail config for json mode |
ApsaraFileConfigDetail (logstoreName, …[, …]) |
The logtail config for Apsara mode |
SyslogConfigDetail (logstoreName, configName, tag) |
The logtail config for syslog mode |
MachineGroupDetail ([group_name, …]) |
The machine group detail info |
IndexConfig ([ttl, line_config, …]) |
The index config of a logstore |
OssShipperConfig (oss_bucket, oss_prefix, …) |
A oss ship config |
OdpsShipperConfig (odps_endpoint, …[, …]) |
Odps shipper config |
ShipperTask (task_id, task_status, …) |
A shipper task |
Project¶
list_project ([offset, size]) |
list the project Unsuccessful opertaion will cause an LogException. |
create_project (project_name, project_des) |
Create a project Unsuccessful opertaion will cause an LogException. |
get_project (project_name) |
get project Unsuccessful opertaion will cause an LogException. |
delete_project (project_name) |
delete project Unsuccessful opertaion will cause an LogException. |
copy_project (from_project, to_project[, …]) |
copy project, logstore, machine group and logtail config to target project, expecting the target project doesn’t contain same named logstores as source project |
Logstore¶
copy_logstore (from_project, from_logstore, …) |
copy logstore, index, logtail config to target logstore, machine group are not included yet. |
list_logstore (project_name[, …]) |
list the logstore in a projectListLogStoreResponse Unsuccessful opertaion will cause an LogException. |
create_logstore (project_name, logstore_name) |
create log store Unsuccessful opertaion will cause an LogException. |
get_logstore (project_name, logstore_name) |
get the logstore meta info Unsuccessful opertaion will cause an LogException. |
update_logstore (project_name, logstore_name) |
update the logstore meta info Unsuccessful opertaion will cause an LogException. |
delete_logstore (project_name, logstore_name) |
delete log store Unsuccessful opertaion will cause an LogException. |
list_topics (request) |
List all topics in a logstore. |
Index¶
create_index (project_name, logstore_name, …) |
create index for a logstore Unsuccessful opertaion will cause an LogException. |
update_index (project_name, logstore_name, …) |
update index for a logstore Unsuccessful opertaion will cause an LogException. |
delete_index (project_name, logstore_name) |
delete index of a logstore Unsuccessful opertaion will cause an LogException. |
get_index_config (project_name, logstore_name) |
get index config detail of a logstore Unsuccessful opertaion will cause an LogException. |
Logtail Config¶
create_logtail_config (project_name, …) |
create logtail config in a project Unsuccessful opertaion will cause an LogException. |
update_logtail_config (project_name, …) |
update logtail config in a project Unsuccessful opertaion will cause an LogException. |
delete_logtail_config (project_name, config_name) |
delete logtail config in a project Unsuccessful opertaion will cause an LogException. |
get_logtail_config (project_name, config_name) |
get logtail config in a project Unsuccessful opertaion will cause an LogException. |
list_logtail_config (project_name[, offset, size]) |
list logtail config name in a project Unsuccessful opertaion will cause an LogException. |
Machine Group¶
create_machine_group (project_name, group_detail) |
create machine group in a project Unsuccessful opertaion will cause an LogException. |
delete_machine_group (project_name, group_name) |
delete machine group in a project Unsuccessful opertaion will cause an LogException. |
update_machine_group (project_name, group_detail) |
update machine group in a project Unsuccessful opertaion will cause an LogException. |
get_machine_group (project_name, group_name) |
get machine group in a project Unsuccessful opertaion will cause an LogException. |
list_machine_group (project_name[, offset, size]) |
list machine group names in a project Unsuccessful opertaion will cause an LogException. |
list_machines (project_name, group_name[, …]) |
list machines in a machine group Unsuccessful opertaion will cause an LogException. |
Apply Logtail Config¶
apply_config_to_machine_group (project_name, …) |
apply a logtail config to a machine group Unsuccessful opertaion will cause an LogException. |
remove_config_to_machine_group (project_name, …) |
remove a logtail config to a machine group Unsuccessful opertaion will cause an LogException. |
get_machine_group_applied_configs (…) |
get the logtail config names applied in a machine group Unsuccessful opertaion will cause an LogException. |
get_config_applied_machine_groups (…) |
get machine group names where the logtail config applies to Unsuccessful opertaion will cause an LogException. |
Shard¶
list_shards (project_name, logstore_name) |
list the shard meta of a logstore Unsuccessful opertaion will cause an LogException. |
split_shard (project_name, logstore_name, …) |
split a readwrite shard into two shards Unsuccessful opertaion will cause an LogException. |
merge_shard (project_name, logstore_name, shardId) |
split two adjacent readwrite hards into one shards Unsuccessful opertaion will cause an LogException. |
Cursor¶
get_cursor (project_name, logstore_name, …) |
Get cursor from log service for batch pull logs Unsuccessful opertaion will cause an LogException. |
get_cursor_time (project_name, logstore_name, …) |
Get cursor time from log service Unsuccessful opertaion will cause an LogException. |
get_previous_cursor_time (project_name, …) |
Get previous cursor time from log service. |
get_begin_cursor (project_name, …) |
Get begin cursor from log service for batch pull logs Unsuccessful opertaion will cause an LogException. |
get_end_cursor (project_name, logstore_name, …) |
Get end cursor from log service for batch pull logs Unsuccessful opertaion will cause an LogException. |
Logs¶
put_logs (request) |
Put logs to log service. |
pull_logs (project_name, logstore_name, …) |
batch pull log data from log service Unsuccessful opertaion will cause an LogException. |
pull_log (project_name, logstore_name, …[, …]) |
batch pull log data from log service using time-range Unsuccessful opertaion will cause an LogException. |
pull_log_dump (project_name, logstore_name, …) |
dump all logs seperatedly line into file_path, file_path, the time parameters are log received time on server side. |
get_log (project, logstore, from_time, to_time) |
Get logs from log service. |
get_logs (request) |
Get logs from log service. |
get_log_all (project, logstore, from_time, …) |
Get logs from log service. |
get_histograms (request) |
Get histograms of requested query from log service. |
get_project_logs (request) |
Get logs from log service. |
Consumer group¶
create_consumer_group (project, logstore, …) |
create consumer group |
update_consumer_group (project, logstore, …) |
Update consumer group |
delete_consumer_group (project, logstore, …) |
Delete consumer group |
list_consumer_group (project, logstore) |
List consumer group |
update_check_point (project, logstore, …[, …]) |
Update check point |
get_check_point (project, logstore, …[, shard]) |
Get check point |
Dashboard¶
create_dashboard (project, detail) |
Create Dashboard. |
update_dashboard (project, detail) |
Update Dashboard. |
delete_dashboard (project, entity) |
Delete Dashboard. |
get_dashboard (project, entity) |
Get Dashboard. |
list_dashboard (project[, offset, size]) |
list the Dashboard, get first 100 items by default Unsuccessful opertaion will cause an LogException. |
Alert¶
create_alert (project, detail) |
Create Alert. |
update_alert (project, detail) |
Update Alert. |
delete_alert (project, entity) |
Delete Alert. |
get_alert (project, entity) |
Get Alert. |
list_alert (project[, offset, size]) |
list the Alert, get first 100 items by default Unsuccessful opertaion will cause an LogException. |
Savedsearch¶
create_savedsearch (project, detail) |
Create Savedsearch. |
update_savedsearch (project, detail) |
Update Savedsearch. |
delete_savedsearch (project, entity) |
Delete Savedsearch. |
get_savedsearch (project, entity) |
Get Savedsearch. |
list_savedsearch (project[, offset, size]) |
list the Savedsearch, get first 100 items by default Unsuccessful opertaion will cause an LogException. |
Shipper¶
create_shipper (project_name, logstore_name, …) |
create odps/oss shipper for every type, it only allowed one shipper Unsuccessful opertaion will cause an LogException. |
update_shipper (project_name, logstore_name, …) |
update odps/oss shipper for every type, it only allowed one shipper Unsuccessful opertaion will cause an LogException. |
delete_shipper (project_name, logstore_name, …) |
delete odps/oss shipper Unsuccessful opertaion will cause an LogException. |
get_shipper_config (project_name, …) |
get odps/oss shipper Unsuccessful opertaion will cause an LogException. |
list_shipper (project_name, logstore_name) |
list odps/oss shipper Unsuccessful opertaion will cause an LogException. |
get_shipper_tasks (project_name, …[, …]) |
get odps/oss shipper tasks in a certain time range Unsuccessful opertaion will cause an LogException. |
retry_shipper_tasks (project_name, …) |
retry failed tasks , only the failed task can be retried Unsuccessful opertaion will cause an LogException. |
ES Migration¶
MigrationManager ([hosts, indexes, query, …]) |
MigrationManager, migrate data from elasticsearch to aliyun log service |
Definitions¶
-
class
aliyun.log.
LogClient
(endpoint, accessKeyId, accessKey, securityToken=None, source=None)[source]¶ Construct the LogClient with endpoint, accessKeyId, accessKey.
Parameters: - endpoint (string) – log service host name, for example, ch-hangzhou.log.aliyuncs.com or https://cn-beijing.log.aliyuncs.com
- accessKeyId (string) – aliyun accessKeyId
- accessKey (string) – aliyun accessKey
-
apply_config_to_machine_group
(project_name, config_name, group_name)[source]¶ apply a logtail config to a machine group Unsuccessful opertaion will cause an LogException.
Parameters: - project_name (string) – the Project name
- config_name (string) – the logtail config name to apply
- group_name (string) – the machine group name
Returns: ApplyConfigToMachineGroupResponse
Raise: LogException
-
arrange_shard
(project, logstore, count)[source]¶ arrange shard to the expected read-write count to a larger one.
Parameters: - project (string) – project name
- logstore (string) – logstore name
- count (int) – expected read-write shard count. should be larger than the current one.
Returns: ’‘
Raise: LogException
-
copy_data
(project, logstore, from_time, to_time=None, to_client=None, to_project=None, to_logstore=None, shard_list=None, batch_size=500, compress=True, new_topic=None, new_source=None)[source]¶ copy data from one logstore to another one (could be the same or in different region), the time is log received time on server side.
Parameters: - project (string) – project name
- logstore (string) – logstore name
- from_time (string/int) – curosr value, could be begin, timestamp or readable time in readable time like “%Y-%m-%d %H:%M:%S CST” e.g. “2018-01-02 12:12:10 CST”, also support human readable string, e.g. “1 hour ago”, “now”, “yesterday 0:0:0”, refer to https://aliyun-log-cli.readthedocs.io/en/latest/tutorials/tutorial_human_readable_datetime.html
- to_time (string/int) – curosr value, default is “end”, could be begin, timestamp or readable time in readable time like “%Y-%m-%d %H:%M:%S CST” e.g. “2018-01-02 12:12:10 CST”, also support human readable string, e.g. “1 hour ago”, “now”, “yesterday 0:0:0”, refer to https://aliyun-log-cli.readthedocs.io/en/latest/tutorials/tutorial_human_readable_datetime.html
- to_client (LogClient) – logclient instance, if empty will use source client
- to_project (string) – project name, if empty will use source project
- to_logstore (string) – logstore name, if empty will use source logstore
- shard_list (string) – shard number list. could be comma seperated list or range: 1,20,31-40
- batch_size (int) – batch size to fetch the data in each iteration. by default it’s 500
- compress (bool) – if use compression, by default it’s True
- new_topic (string) – overwrite the copied topic with the passed one
- new_source (string) – overwrite the copied source with the passed one
Returns: LogResponse {“total_count”: 30, “shards”: {0: 10, 1: 20} })
-
copy_logstore
(from_project, from_logstore, to_logstore, to_project=None, to_client=None)[source]¶ copy logstore, index, logtail config to target logstore, machine group are not included yet. the target logstore will be crated if not existing
Parameters: - from_project (string) – project name
- from_logstore (string) – logstore name
- to_logstore (string) – target logstore name
- to_project (string) – target project name, copy to same project if not being specified, will try to create it if not being specified
- to_client (LogClient) – logclient instance, use it to operate on the “to_project” if being specified for cross region purpose
Returns:
-
copy_project
(from_project, to_project, to_client=None, copy_machine_group=False)[source]¶ copy project, logstore, machine group and logtail config to target project, expecting the target project doesn’t contain same named logstores as source project
Parameters: - from_project (string) – project name
- to_project (string) – project name
- to_client (LogClient) – logclient instance
- copy_machine_group (bool) – if copy machine group resources, False by default.
Returns: None
-
create_alert
(project, detail)¶ Create Alert. Unsuccessful opertaion will cause an LogException.
Parameters: - project (string) – project name
- detail (dict/string) – json string
Returns: CreateEntityResponse
Raise: LogException
-
create_consumer_group
(project, logstore, consumer_group, timeout, in_order=False)[source]¶ create consumer group
Parameters: - project (string) – project name
- logstore (string) – logstore name
- consumer_group (string) – consumer group name
- timeout (int) – time-out
- in_order (bool) – if consume in order, default is False
Returns: CreateConsumerGroupResponse
-
create_dashboard
(project, detail)¶ Create Dashboard. Unsuccessful opertaion will cause an LogException.
Parameters: - project (string) – project name
- detail (dict/string) – json string
Returns: CreateEntityResponse
Raise: LogException
-
create_external_store
(project_name, config)[source]¶ create log store Unsuccessful opertaion will cause an LogException.
Parameters: project_name (string) – the Project name :type config : ExternalStoreConfig :param config :external store config
Returns: CreateExternalStoreResponse Raise: LogException
-
create_index
(project_name, logstore_name, index_detail)[source]¶ create index for a logstore Unsuccessful opertaion will cause an LogException.
Parameters: - project_name (string) – the Project name
- logstore_name (string) – the logstore name
- index_detail (IndexConfig) – the index config detail used to create index
Returns: CreateIndexResponse
Raise: LogException
-
create_logstore
(project_name, logstore_name, ttl=30, shard_count=2, enable_tracking=False, append_meta=False, auto_split=True, max_split_shard=64, preserve_storage=False)[source]¶ create log store Unsuccessful opertaion will cause an LogException.
Parameters: - project_name (string) – the Project name
- logstore_name (string) – the logstore name
- ttl (int) – the life cycle of log in the logstore in days, default 30, up to 3650
- shard_count (int) – the shard count of the logstore to create, default 2
- enable_tracking (bool) – enable web tracking, default is False
- append_meta (bool) – allow to append meta info (server received time and IP for external IP to each received log)
- auto_split (bool) – auto split shard, max_split_shard will be 64 by default is True
- max_split_shard (int) – max shard to split, up to 64
- preserve_storage (bool) – if always persist data, TTL will be ignored.
Returns: CreateLogStoreResponse
Raise: LogException
-
create_logtail_config
(project_name, config_detail)[source]¶ create logtail config in a project Unsuccessful opertaion will cause an LogException.
Parameters: - project_name (string) – the Project name
- config_detail (LogtailConfigGenerator or SeperatorFileConfigDetail or SimpleFileConfigDetail or FullRegFileConfigDetail or JsonFileConfigDetail or ApsaraFileConfigDetail or SyslogConfigDetail or CommonRegLogConfigDetail) – the logtail config detail info, use LogtailConfigGenerator.from_json to generate config: SeperatorFileConfigDetail or SimpleFileConfigDetail or FullRegFileConfigDetail or JsonFileConfigDetail or ApsaraFileConfigDetail or SyslogConfigDetail, Note: CommonRegLogConfigDetail is deprecated.
Returns: CreateLogtailConfigResponse
Raise: LogException
-
create_machine_group
(project_name, group_detail)[source]¶ create machine group in a project Unsuccessful opertaion will cause an LogException.
Parameters: - project_name (string) – the Project name
- group_detail (MachineGroupDetail) – the machine group detail config
Returns: CreateMachineGroupResponse
Raise: LogException
-
create_project
(project_name, project_des)[source]¶ Create a project Unsuccessful opertaion will cause an LogException.
Parameters: - project_name (string) – the Project name
- project_des (string) – the description of a project
Returns: CreateProjectResponse
Raise: LogException
-
create_savedsearch
(project, detail)¶ Create Savedsearch. Unsuccessful opertaion will cause an LogException.
Parameters: - project (string) – project name
- detail (dict/string) – json string
Returns: CreateEntityResponse
Raise: LogException
-
create_shipper
(project_name, logstore_name, shipper_name, shipper_type, shipper_config)[source]¶ create odps/oss shipper for every type, it only allowed one shipper Unsuccessful opertaion will cause an LogException.
Parameters: - project_name (string) – the Project name
- logstore_name (string) – the logstore name
- shipper_name (string) – the shipper name
- shipper_type (string) – only support “odps” or “oss”
- shipper_config (OssShipperConfig or OdpsShipperConfig) – the detail shipper config, must be OssShipperConfig or OdpsShipperConfig type
Returns: CreateShipperResponse
Raise: LogException
-
delete_alert
(project, entity)¶ Delete Alert. Unsuccessful opertaion will cause an LogException.
Parameters: - project (string) – project name
- entity (string) – alert name
Returns: DeleteEntityResponse
Raise: LogException
-
delete_consumer_group
(project, logstore, consumer_group)[source]¶ Delete consumer group
Parameters: - project (string) – project name
- logstore (string) – logstore name
- consumer_group (string) – consumer group name
Returns: None
-
delete_dashboard
(project, entity)¶ Delete Dashboard. Unsuccessful opertaion will cause an LogException.
Parameters: - project (string) – project name
- entity (string) – dashboard name
Returns: DeleteEntityResponse
Raise: LogException
-
delete_external_store
(project_name, store_name)[source]¶ delete log store Unsuccessful opertaion will cause an LogException.
Parameters: - project_name (string) – the Project name
- store_name (string) – the external store name
Returns: DeleteExternalStoreResponse
Raise: LogException
-
delete_index
(project_name, logstore_name)[source]¶ delete index of a logstore Unsuccessful opertaion will cause an LogException.
Parameters: - project_name (string) – the Project name
- logstore_name (string) – the logstore name
Returns: DeleteIndexResponse
Raise: LogException
-
delete_logstore
(project_name, logstore_name)[source]¶ delete log store Unsuccessful opertaion will cause an LogException.
Parameters: - project_name (string) – the Project name
- logstore_name (string) – the logstore name
Returns: DeleteLogStoreResponse
Raise: LogException
-
delete_logtail_config
(project_name, config_name)[source]¶ delete logtail config in a project Unsuccessful opertaion will cause an LogException.
Parameters: - project_name (string) – the Project name
- config_name (string) – the logtail config name
Returns: DeleteLogtailConfigResponse
Raise: LogException
-
delete_machine_group
(project_name, group_name)[source]¶ delete machine group in a project Unsuccessful opertaion will cause an LogException.
Parameters: - project_name (string) – the Project name
- group_name (string) – the group name
Returns: DeleteMachineGroupResponse
Raise: LogException
-
delete_project
(project_name)[source]¶ delete project Unsuccessful opertaion will cause an LogException.
Parameters: project_name (string) – the Project name Returns: DeleteProjectResponse Raise: LogException
-
delete_savedsearch
(project, entity)¶ Delete Savedsearch. Unsuccessful opertaion will cause an LogException.
Parameters: - project (string) – project name
- entity (string) – savedsearch name
Returns: DeleteEntityResponse
Raise: LogException
-
delete_shard
(project_name, logstore_name, shardId)[source]¶ delete a readonly shard Unsuccessful opertaion will cause an LogException.
Parameters: - project_name (string) – the Project name
- logstore_name (string) – the logstore name
- shardId (int) – the read only shard id
Returns: ListShardResponse
Raise: LogException
-
delete_shipper
(project_name, logstore_name, shipper_name)[source]¶ delete odps/oss shipper Unsuccessful opertaion will cause an LogException.
Parameters: - project_name (string) – the Project name
- logstore_name (string) – the logstore name
- shipper_name (string) – the shipper name
Returns: DeleteShipperResponse
Raise: LogException
-
es_migration
(hosts, project_name, indexes=None, query=None, scroll='5m', logstore_index_mappings=None, pool_size=10, time_reference=None, source=None, topic=None, wait_time_in_secs=60, auto_creation=True)[source]¶ migrate data from elasticsearch to aliyun log service
Parameters: - hosts (string) – a comma-separated list of source ES nodes. e.g. “localhost:9200,other_host:9200”
- project_name (string) – specify the project_name of your log services. e.g. “your_project”
- indexes (string) – a comma-separated list of source index names. e.g. “index1,index2”
- query (string) – used to filter docs, so that you can specify the docs you want to migrate. e.g. ‘{“query”: {“match”: {“title”: “python”}}}’
- scroll (string) – specify how long a consistent view of the index should be maintained for scrolled search. e.g. “5m”
- logstore_index_mappings (string) – specify the mappings of log service logstore and ES index. e.g. ‘{“logstore1”: “my_index*”, “logstore2”: “index1,index2”}, “logstore3”: “index3”}’
- pool_size (int) – specify the size of process pool. e.g. 10
- time_reference (string) – specify what ES doc’s field to use as log’s time field. e.g. “field1”
- source (string) – specify the value of log’s source field. e.g. “your_source”
- topic (string) – specify the value of log’s topic field. e.g. “your_topic”
- wait_time_in_secs (int) – specify the waiting time between initialize aliyun log and executing data migration task. e.g. 60
- auto_creation (bool) – specify whether to let the tool create logstore and index automatically for you. e.g. True
Returns: MigrationResponse
Raise: Exception
-
get_alert
(project, entity)¶ Get Alert. Unsuccessful opertaion will cause an LogException.
Parameters: - project (string) – project name
- entity (string) – alert name
Returns: GetEntityResponse
Raise: LogException
-
get_begin_cursor
(project_name, logstore_name, shard_id)[source]¶ Get begin cursor from log service for batch pull logs Unsuccessful opertaion will cause an LogException.
Parameters: - project_name (string) – the Project name
- logstore_name (string) – the logstore name
- shard_id (int) – the shard id
Returns: GetLogsResponse
Raise: LogException
-
get_check_point
(project, logstore, consumer_group, shard=-1)[source]¶ Get check point
Parameters: - project (string) – project name
- logstore (string) – logstore name
- consumer_group (string) – consumer group name
- shard (int) – shard id
Returns: ConsumerGroupCheckPointResponse
-
get_check_point_fixed
(project, logstore, consumer_group, shard=-1)[source]¶ Get check point
Parameters: - project (string) – project name
- logstore (string) – logstore name
- consumer_group (string) – consumer group name
- shard (int) – shard id
Returns: ConsumerGroupCheckPointResponse
-
get_config_applied_machine_groups
(project_name, config_name)[source]¶ get machine group names where the logtail config applies to Unsuccessful opertaion will cause an LogException.
Parameters: - project_name (string) – the Project name
- config_name (string) – the logtail config name used to apply
Returns: GetConfigAppliedMachineGroupsResponse
Raise: LogException
-
get_cursor
(project_name, logstore_name, shard_id, start_time)[source]¶ Get cursor from log service for batch pull logs Unsuccessful opertaion will cause an LogException.
Parameters: - project_name (string) – the Project name
- logstore_name (string) – the logstore name
- shard_id (int) – the shard id
- start_time (string/int) – the start time of cursor, e.g 1441093445 or “begin”/”end”, or readable time like “%Y-%m-%d %H:%M:%S CST”, also support human readable string, e.g. “1 hour ago”, “now”, “yesterday 0:0:0”, refer to https://aliyun-log-cli.readthedocs.io/en/latest/tutorials/tutorial_human_readable_datetime.html
Returns: GetCursorResponse
Raise: LogException
-
get_cursor_time
(project_name, logstore_name, shard_id, cursor)[source]¶ Get cursor time from log service Unsuccessful opertaion will cause an LogException.
Parameters: - project_name (string) – the Project name
- logstore_name (string) – the logstore name
- shard_id (int) – the shard id
- cursor (string) – the cursor to get its service receive time
Returns: GetCursorTimeResponse
Raise: LogException
-
get_dashboard
(project, entity)¶ Get Dashboard. Unsuccessful opertaion will cause an LogException.
Parameters: - project (string) – project name
- entity (string) – dashboard name
Returns: GetEntityResponse
Raise: LogException
-
get_end_cursor
(project_name, logstore_name, shard_id)[source]¶ Get end cursor from log service for batch pull logs Unsuccessful opertaion will cause an LogException.
Parameters: - project_name (string) – the Project name
- logstore_name (string) – the logstore name
- shard_id (int) – the shard id
Returns: GetLogsResponse
Raise: LogException
-
get_external_store
(project_name, store_name)[source]¶ get the logstore meta info Unsuccessful opertaion will cause an LogException.
Parameters: - project_name (string) – the Project name
- store_name (string) – the logstore name
Returns: GetLogStoreResponse
Raise: LogException
-
get_histograms
(request)[source]¶ Get histograms of requested query from log service. Unsuccessful opertaion will cause an LogException.
Parameters: request (GetHistogramsRequest) – the GetHistograms request parameters class. Returns: GetHistogramsResponse Raise: LogException
-
get_index_config
(project_name, logstore_name)[source]¶ get index config detail of a logstore Unsuccessful opertaion will cause an LogException.
Parameters: - project_name (string) – the Project name
- logstore_name (string) – the logstore name
Returns: GetIndexResponse
Raise: LogException
-
get_log
(project, logstore, from_time, to_time, topic=None, query=None, reverse=False, offset=0, size=100)[source]¶ Get logs from log service. will retry when incomplete. Unsuccessful opertaion will cause an LogException. Note: for larger volume of data (e.g. > 1 million logs), use get_log_all
Parameters: - project (string) – project name
- logstore (string) – logstore name
- from_time (int/string) – the begin timestamp or format of time in readable time like “%Y-%m-%d %H:%M:%S CST” e.g. “2018-01-02 12:12:10 CST”, also support human readable string, e.g. “1 hour ago”, “now”, “yesterday 0:0:0”, refer to https://aliyun-log-cli.readthedocs.io/en/latest/tutorials/tutorial_human_readable_datetime.html
- to_time (int/string) – the end timestamp or format of time in readable time like “%Y-%m-%d %H:%M:%S CST” e.g. “2018-01-02 12:12:10 CST”, also support human readable string, e.g. “1 hour ago”, “now”, “yesterday 0:0:0”, refer to https://aliyun-log-cli.readthedocs.io/en/latest/tutorials/tutorial_human_readable_datetime.html
- topic (string) – topic name of logs, could be None
- query (string) – user defined query, could be None
- reverse (bool) – if reverse is set to true, the query will return the latest logs first, default is false
- offset (int) – line offset of return logs
- size (int) – max line number of return logs, -1 means get all
Returns: GetLogsResponse
Raise: LogException
-
get_log_all
(project, logstore, from_time, to_time, topic=None, query=None, reverse=False, offset=0)[source]¶ Get logs from log service. will retry when incomplete. Unsuccessful opertaion will cause an LogException. different with get_log with size=-1, It will try to iteratively fetch all data every 100 items and yield them, in CLI, it could apply jmes filter to each batch and make it possible to fetch larger volume of data.
Parameters: - project (string) – project name
- logstore (string) – logstore name
- from_time (int/string) – the begin timestamp or format of time in readable time like “%Y-%m-%d %H:%M:%S CST” e.g. “2018-01-02 12:12:10 CST”, also support human readable string, e.g. “1 hour ago”, “now”, “yesterday 0:0:0”, refer to https://aliyun-log-cli.readthedocs.io/en/latest/tutorials/tutorial_human_readable_datetime.html
- to_time (int/string) – the end timestamp or format of time in readable time like “%Y-%m-%d %H:%M:%S CST” e.g. “2018-01-02 12:12:10 CST”, also support human readable string, e.g. “1 hour ago”, “now”, “yesterday 0:0:0”, refer to https://aliyun-log-cli.readthedocs.io/en/latest/tutorials/tutorial_human_readable_datetime.html
- topic (string) – topic name of logs, could be None
- query (string) – user defined query, could be None
- reverse (bool) – if reverse is set to true, the query will return the latest logs first, default is false
- offset (int) – offset to start, by default is 0
Returns: GetLogsResponse iterator
Raise: LogException
-
get_logs
(request)[source]¶ Get logs from log service. Unsuccessful opertaion will cause an LogException. Note: for larger volume of data (e.g. > 1 million logs), use get_log_all
Parameters: request (GetLogsRequest) – the GetLogs request parameters class. Returns: GetLogsResponse Raise: LogException
-
get_logstore
(project_name, logstore_name)[source]¶ get the logstore meta info Unsuccessful opertaion will cause an LogException.
Parameters: - project_name (string) – the Project name
- logstore_name (string) – the logstore name
Returns: GetLogStoreResponse
Raise: LogException
-
get_logtail_config
(project_name, config_name)[source]¶ get logtail config in a project Unsuccessful opertaion will cause an LogException.
Parameters: - project_name (string) – the Project name
- config_name (string) – the logtail config name
Returns: GetLogtailConfigResponse
Raise: LogException
-
get_machine_group
(project_name, group_name)[source]¶ get machine group in a project Unsuccessful opertaion will cause an LogException.
Parameters: - project_name (string) – the Project name
- group_name (string) – the group name to get
Returns: GetMachineGroupResponse
Raise: LogException
-
get_machine_group_applied_configs
(project_name, group_name)[source]¶ get the logtail config names applied in a machine group Unsuccessful opertaion will cause an LogException.
Parameters: - project_name (string) – the Project name
- group_name (string) – the group name list
Returns: GetMachineGroupAppliedConfigResponse
Raise: LogException
-
get_previous_cursor_time
(project_name, logstore_name, shard_id, cursor, normalize=True)[source]¶ Get previous cursor time from log service. Note: normalize = true: if the cursor is out of range, it will be nornalized to nearest cursor Unsuccessful opertaion will cause an LogException.
Parameters: - project_name (string) – the Project name
- logstore_name (string) – the logstore name
- shard_id (int) – the shard id
- cursor (string) – the cursor to get its service receive time
- normalize (bool) – fix the cursor or not if it’s out of scope
Returns: GetCursorTimeResponse
Raise: LogException
-
get_project
(project_name)[source]¶ get project Unsuccessful opertaion will cause an LogException.
Parameters: project_name (string) – the Project name Returns: GetProjectResponse Raise: LogException
-
get_project_logs
(request)[source]¶ Get logs from log service. Unsuccessful opertaion will cause an LogException.
Parameters: request (GetProjectLogsRequest) – the GetProjectLogs request parameters class. Returns: GetLogsResponse Raise: LogException
-
get_resource_usage
(project)[source]¶ get resource usage ist the project Unsuccessful opertaion will cause an LogException.
Parameters: client (string) – project name Returns: dict Raise: LogException
-
get_savedsearch
(project, entity)¶ Get Savedsearch. Unsuccessful opertaion will cause an LogException.
Parameters: - project (string) – project name
- entity (string) – savedsearch name
Returns: GetEntityResponse
Raise: LogException
-
get_shipper_config
(project_name, logstore_name, shipper_name)[source]¶ get odps/oss shipper Unsuccessful opertaion will cause an LogException.
Parameters: - project_name (string) – the Project name
- logstore_name (string) – the logstore name
- shipper_name (string) – the shipper name
Returns: GetShipperConfigResponse
Raise: LogException
-
get_shipper_tasks
(project_name, logstore_name, shipper_name, start_time, end_time, status_type='', offset=0, size=100)[source]¶ get odps/oss shipper tasks in a certain time range Unsuccessful opertaion will cause an LogException.
Parameters: - project_name (string) – the Project name
- logstore_name (string) – the logstore name
- shipper_name (string) – the shipper name
- start_time (int) – the start timestamp
- end_time (int) – the end timestamp
- status_type (string) – support one of [‘’, ‘fail’, ‘success’, ‘running’] , if the status_type = ‘’ , return all kinds of status type
- offset (int) – the begin task offset, -1 means all
- size (int) – the needed tasks count
Returns: GetShipperTasksResponse
Raise: LogException
-
heart_beat
(project, logstore, consumer_group, consumer, shards=None)[source]¶ Heatbeat consumer group
Parameters: - project (string) – project name
- logstore (string) – logstore name
- consumer_group (string) – consumer group name
- consumer (string) – consumer name
- shards (int list) – shard id list e.g. [0,1,2]
Returns: None
-
list_alert
(project, offset=0, size=100)¶ list the Alert, get first 100 items by default Unsuccessful opertaion will cause an LogException.
Parameters: - project (string) – the Project name
- offset (int) – the offset of all the matched names
- size (int) – the max return names count, -1 means all
Returns: ListLogStoreResponse
Raise: LogException
-
list_consumer_group
(project, logstore)[source]¶ List consumer group
Parameters: - project (string) – project name
- logstore (string) – logstore name
Returns: ListConsumerGroupResponse
-
list_dashboard
(project, offset=0, size=100)¶ list the Dashboard, get first 100 items by default Unsuccessful opertaion will cause an LogException.
Parameters: - project (string) – the Project name
- offset (int) – the offset of all the matched names
- size (int) – the max return names count, -1 means all
Returns: ListLogStoreResponse
Raise: LogException
-
list_external_store
(project_name, external_store_name_pattern=None, offset=0, size=100)[source]¶ list the logstore in a projectListLogStoreResponse Unsuccessful opertaion will cause an LogException.
Parameters: - project_name (string) – the Project name
- logstore_name_pattern (string) – the sub name logstore, used for the server to return logstore names contain this sub name
- offset (int) – the offset of all the matched names
- size (int) – the max return names count, -1 means all
Returns: ListLogStoreResponse
Raise: LogException
-
list_logstore
(project_name, logstore_name_pattern=None, offset=0, size=100)[source]¶ list the logstore in a projectListLogStoreResponse Unsuccessful opertaion will cause an LogException.
Parameters: - project_name (string) – the Project name
- logstore_name_pattern (string) – the sub name logstore, used for the server to return logstore names contain this sub name
- offset (int) – the offset of all the matched names
- size (int) – the max return names count, -1 means all
Returns: ListLogStoreResponse
Raise: LogException
-
list_logstore_acl
(project_name, logstore_name, offset=0, size=100)[source]¶ list acl of a logstore Unsuccessful opertaion will cause an LogException.
Parameters: - project_name (string) – the Project name
- logstore_name (string) – the logstore name
- offset (int) – the offset of all acl
- size (int) – the max return acl count
Returns: ListAclResponse
Raise: LogException
-
list_logstores
(request)[source]¶ List all logstores of requested project. Unsuccessful opertaion will cause an LogException.
Parameters: request (ListLogstoresRequest) – the ListLogstores request parameters class. Returns: ListLogStoresResponse Raise: LogException
-
list_logtail_config
(project_name, offset=0, size=100)[source]¶ list logtail config name in a project Unsuccessful opertaion will cause an LogException.
Parameters: - project_name (string) – the Project name
- offset (int) – the offset of all config names
- size (int) – the max return names count, -1 means all
Returns: ListLogtailConfigResponse
Raise: LogException
-
list_machine_group
(project_name, offset=0, size=100)[source]¶ list machine group names in a project Unsuccessful opertaion will cause an LogException.
Parameters: - project_name (string) – the Project name
- offset (int) – the offset of all group name
- size (int) – the max return names count, -1 means all
Returns: ListMachineGroupResponse
Raise: LogException
-
list_machines
(project_name, group_name, offset=0, size=100)[source]¶ list machines in a machine group Unsuccessful opertaion will cause an LogException.
Parameters: - project_name (string) – the Project name
- group_name (string) – the group name to list
- offset (int) – the offset of all group name
- size (int) – the max return names count, -1 means all
Returns: ListMachinesResponse
Raise: LogException
-
list_project
(offset=0, size=100)[source]¶ list the project Unsuccessful opertaion will cause an LogException.
Parameters: - offset (int) – the offset of all the matched names
- size (int) – the max return names count, -1 means return all data
Returns: ListProjectResponse
Raise: LogException
-
list_project_acl
(project_name, offset=0, size=100)[source]¶ list acl of a project Unsuccessful opertaion will cause an LogException.
Parameters: - project_name (string) – the Project name
- offset (int) – the offset of all acl
- size (int) – the max return acl count
Returns: ListAclResponse
Raise: LogException
-
list_savedsearch
(project, offset=0, size=100)¶ list the Savedsearch, get first 100 items by default Unsuccessful opertaion will cause an LogException.
Parameters: - project (string) – the Project name
- offset (int) – the offset of all the matched names
- size (int) – the max return names count, -1 means all
Returns: ListLogStoreResponse
Raise: LogException
-
list_shards
(project_name, logstore_name)[source]¶ list the shard meta of a logstore Unsuccessful opertaion will cause an LogException.
Parameters: - project_name (string) – the Project name
- logstore_name (string) – the logstore name
Returns: ListShardResponse
Raise: LogException
-
list_shipper
(project_name, logstore_name)[source]¶ list odps/oss shipper Unsuccessful opertaion will cause an LogException.
Parameters: - project_name (string) – the Project name
- logstore_name (string) – the logstore name
Returns: ListShipperResponse
Raise: LogException
-
list_topics
(request)[source]¶ List all topics in a logstore. Unsuccessful opertaion will cause an LogException.
Parameters: request (ListTopicsRequest) – the ListTopics request parameters class. Returns: ListTopicsResponse Raise: LogException
-
merge_shard
(project_name, logstore_name, shardId)[source]¶ split two adjacent readwrite hards into one shards Unsuccessful opertaion will cause an LogException.
Parameters: - project_name (string) – the Project name
- logstore_name (string) – the logstore name
- shardId (int) – the shard id of the left shard, server will determine the right adjacent shardId
Returns: ListShardResponse
Raise: LogException
-
pull_log
(project_name, logstore_name, shard_id, from_time, to_time, batch_size=1000, compress=True)[source]¶ batch pull log data from log service using time-range Unsuccessful opertaion will cause an LogException. the time parameter means the time when server receives the logs
Parameters: - project_name (string) – the Project name
- logstore_name (string) – the logstore name
- shard_id (int) – the shard id
- from_time (string/int) – curosr value, could be begin, timestamp or readable time in readable time like “%Y-%m-%d %H:%M:%S CST” e.g. “2018-01-02 12:12:10 CST”, also support human readable string, e.g. “1 hour ago”, “now”, “yesterday 0:0:0”, refer to https://aliyun-log-cli.readthedocs.io/en/latest/tutorials/tutorial_human_readable_datetime.html
- to_time (string/int) – curosr value, could be begin, timestamp or readable time in readable time like “%Y-%m-%d %H:%M:%S CST” e.g. “2018-01-02 12:12:10 CST”, also support human readable string, e.g. “1 hour ago”, “now”, “yesterday 0:0:0”, refer to https://aliyun-log-cli.readthedocs.io/en/latest/tutorials/tutorial_human_readable_datetime.html
- batch_size (int) – batch size to fetch the data in each iteration. by default it’s 1000
- compress (bool) – if use compression, by default it’s True
Returns: PullLogResponse
Raise: LogException
-
pull_log_dump
(project_name, logstore_name, from_time, to_time, file_path, batch_size=500, compress=True, encodings=None)[source]¶ dump all logs seperatedly line into file_path, file_path, the time parameters are log received time on server side.
Parameters: - project_name (string) – the Project name
- logstore_name (string) – the logstore name
- from_time (string/int) – curosr value, could be begin, timestamp or readable time in readable time like “%Y-%m-%d %H:%M:%S CST” e.g. “2018-01-02 12:12:10 CST”, also support human readable string, e.g. “1 hour ago”, “now”, “yesterday 0:0:0”, refer to https://aliyun-log-cli.readthedocs.io/en/latest/tutorials/tutorial_human_readable_datetime.html
- to_time (string/int) – curosr value, could be begin, timestamp or readable time in readable time like “%Y-%m-%d %H:%M:%S CST” e.g. “2018-01-02 12:12:10 CST”, also support human readable string, e.g. “1 hour ago”, “now”, “yesterday 0:0:0”, refer to https://aliyun-log-cli.readthedocs.io/en/latest/tutorials/tutorial_human_readable_datetime.html
- file_path (string) – file path with {} for shard id. e.g. “/data/dump_{}.data”, {} will be replaced with each partition.
- batch_size (int) – batch size to fetch the data in each iteration. by default it’s 500
- compress (bool) – if use compression, by default it’s True
- encodings (string list) – encoding like [“utf8”, “latin1”] etc to dumps the logs in json format to file. default is [“utf8”,]
Returns: LogResponse {“total_count”: 30, “files”: {‘file_path_1’: 10, “file_path_2”: 20} })
Raise: LogException
-
pull_logs
(project_name, logstore_name, shard_id, cursor, count=1000, end_cursor=None, compress=True)[source]¶ batch pull log data from log service Unsuccessful opertaion will cause an LogException.
Parameters: - project_name (string) – the Project name
- logstore_name (string) – the logstore name
- shard_id (int) – the shard id
- cursor (string) – the start to cursor to get data
- count (int) – the required pull log package count, default 1000 packages
- end_cursor (string) – the end cursor position to get data
- compress (boolean) – if use zip compress for transfer data
Returns: PullLogResponse
Raise: LogException
-
put_log_raw
(project, logstore, log_group, compress=None)[source]¶ Put logs to log service. using raw data in protobuf
Parameters: - project (string) – the Project name
- logstore (string) – the logstore name
- log_group (LogGroup) – log group structure
- compress (boolean) – compress or not, by default is True
Returns: PutLogsResponse
Raise: LogException
-
put_logs
(request)[source]¶ Put logs to log service. up to 512000 logs up to 10MB size Unsuccessful opertaion will cause an LogException.
Parameters: request (PutLogsRequest) – the PutLogs request parameters class Returns: PutLogsResponse Raise: LogException
-
remove_config_to_machine_group
(project_name, config_name, group_name)[source]¶ remove a logtail config to a machine group Unsuccessful opertaion will cause an LogException.
Parameters: - project_name (string) – the Project name
- config_name (string) – the logtail config name to apply
- group_name (string) – the machine group name
Returns: RemoveConfigToMachineGroupResponse
Raise: LogException
-
retry_shipper_tasks
(project_name, logstore_name, shipper_name, task_list)[source]¶ retry failed tasks , only the failed task can be retried Unsuccessful opertaion will cause an LogException.
Parameters: - project_name (string) – the Project name
- logstore_name (string) – the logstore name
- shipper_name (string) – the shipper name
- task_list (string array) – the failed task_id list, e.g [‘failed_task_id_1’, ‘failed_task_id_2’,…], currently the max retry task count 10 every time
Returns: RetryShipperTasksResponse
Raise: LogException
-
set_source
(source)[source]¶ Set the source of the log client
Parameters: source (string) – new source Returns: None
-
set_user_agent
(user_agent)[source]¶ set user agent
Parameters: user_agent (string) – user agent Returns: None
-
split_shard
(project_name, logstore_name, shardId, split_hash)[source]¶ split a readwrite shard into two shards Unsuccessful opertaion will cause an LogException.
Parameters: - project_name (string) – the Project name
- logstore_name (string) – the logstore name
- shardId (int) – the shard id
- split_hash (string) – the internal hash between the shard begin and end hash
Returns: ListShardResponse
Raise: LogException
-
transform_data
(project, logstore, config, from_time, to_time=None, to_client=None, to_project=None, to_logstore=None, shard_list=None, batch_size=500, compress=True, cg_name=None, c_name=None, cg_heartbeat_interval=None, cg_data_fetch_interval=None, cg_in_order=None, cg_worker_pool_size=None)[source]¶ transform data from one logstore to another one (could be the same or in different region), the time passed is log received time on server side. There’re two mode, batch mode / consumer group mode. For Batch mode, just leave the cg_name and later options as None.
Parameters: - project (string) – project name
- logstore (string) – logstore name
- config (string) – transform config imported or path of config (in python)
- from_time (string/int) – curosr value, could be begin, timestamp or readable time in readable time like “%Y-%m-%d %H:%M:%S CST” e.g. “2018-01-02 12:12:10 CST”, also support human readable string, e.g. “1 hour ago”, “now”, “yesterday 0:0:0”, refer to https://aliyun-log-cli.readthedocs.io/en/latest/tutorials/tutorial_human_readable_datetime.html
- to_time (string/int) – curosr value, leave it as None if consumer group is configured. could be begin, timestamp or readable time in readable time like “%Y-%m-%d %H:%M:%S CST” e.g. “2018-01-02 12:12:10 CST”, also support human readable string, e.g. “1 hour ago”, “now”, “yesterday 0:0:0”, refer to https://aliyun-log-cli.readthedocs.io/en/latest/tutorials/tutorial_human_readable_datetime.html
- to_client (LogClient) – logclient instance, if empty will use source client
- to_project (string) – project name, if empty will use source project
- to_logstore (string) – logstore name, if empty will use source logstore
- shard_list (string) – shard number list. could be comma seperated list or range: 1,20,31-40
- batch_size (int) – batch size to fetch the data in each iteration. by default it’s 500
- compress (bool) – if use compression, by default it’s True
- cg_name (string) – consumer group name. must configure if it’s consumer group mode.
- c_name (string) – consumer group name for consumer group mode, default: CLI-transform-data-${process_id}
- cg_heartbeat_interval (int) – cg_heartbeat_interval, default 20
- cg_data_fetch_interval (int) – cg_data_fetch_interval, default 2
- cg_in_order (bool) – cg_in_order, default False
- cg_worker_pool_size (int) – cg_worker_pool_size, default 2
Returns: LogResponse {“total_count”: 30, “shards”: {0: {“count”: 10, “removed”: 1}, 2: {“count”: 20, “removed”: 1}} })
-
update_alert
(project, detail)¶ Update Alert. Unsuccessful opertaion will cause an LogException.
Parameters: - project (string) – project name
- detail (dict/string) – json string
Returns: UpdateEntityResponse
Raise: LogException
-
update_check_point
(project, logstore, consumer_group, shard, check_point, consumer='', force_success=True)[source]¶ Update check point
Parameters: - project (string) – project name
- logstore (string) – logstore name
- consumer_group (string) – consumer group name
- shard (int) – shard id
- check_point (string) – checkpoint name
- consumer (string) – consumer name
- force_success (bool) – if force to succeed
Returns: None
-
update_consumer_group
(project, logstore, consumer_group, timeout=None, in_order=None)[source]¶ Update consumer group
Parameters: - project (string) – project name
- logstore (string) – logstore name
- consumer_group (string) – consumer group name
- timeout (int) – timeout
- in_order (bool) – order
Returns: None
-
update_dashboard
(project, detail)¶ Update Dashboard. Unsuccessful opertaion will cause an LogException.
Parameters: - project (string) – project name
- detail (dict/string) – json string
Returns: UpdateEntityResponse
Raise: LogException
-
update_external_store
(project_name, config)[source]¶ update the logstore meta info Unsuccessful opertaion will cause an LogException.
:param config : external store config
Returns: UpdateExternalStoreResponse Raise: LogException
-
update_index
(project_name, logstore_name, index_detail)[source]¶ update index for a logstore Unsuccessful opertaion will cause an LogException.
Parameters: - project_name (string) – the Project name
- logstore_name (string) – the logstore name
- index_detail (IndexConfig) – the index config detail used to update index
Returns: UpdateIndexResponse
Raise: LogException
-
update_logstore
(project_name, logstore_name, ttl=None, enable_tracking=None, shard_count=None, append_meta=None, auto_split=None, max_split_shard=None, preserve_storage=None)[source]¶ update the logstore meta info Unsuccessful opertaion will cause an LogException.
Parameters: - project_name (string) – the Project name
- logstore_name (string) – the logstore name
- ttl (int) – the life cycle of log in the logstore in days
- enable_tracking (bool) – enable web tracking
- shard_count (int) – deprecated, the shard count could only be updated by split & merge
- append_meta (bool) – allow to append meta info (server received time and IP for external IP to each received log)
- auto_split (bool) – auto split shard, max_split_shard will be 64 by default is True
- max_split_shard (int) – max shard to split, up to 64
- preserve_storage (bool) – if always persist data, TTL will be ignored.
Returns: UpdateLogStoreResponse
Raise: LogException
-
update_logstore_acl
(project_name, logstore_name, acl_action, acl_config)[source]¶ update acl of a logstore Unsuccessful opertaion will cause an LogException.
Parameters: - project_name (string) – the Project name
- logstore_name (string) – the logstore name
- acl_action (string) – “grant” or “revoke”, grant or revoke the acl_config to/from a logstore
- acl_config (acl_config.AclConfig) – the detail acl config info
Returns: UpdateAclResponse
Raise: LogException
-
update_logtail_config
(project_name, config_detail)[source]¶ update logtail config in a project Unsuccessful opertaion will cause an LogException.
Parameters: - project_name (string) – the Project name
- config_detail (LogtailConfigGenerator or SeperatorFileConfigDetail or SimpleFileConfigDetail or FullRegFileConfigDetail or JsonFileConfigDetail or ApsaraFileConfigDetail or SyslogConfigDetail or CommonRegLogConfigDetail) – the logtail config detail info, use LogtailConfigGenerator.from_json to generate config: SeperatorFileConfigDetail or SimpleFileConfigDetail or FullRegFileConfigDetail or JsonFileConfigDetail or ApsaraFileConfigDetail or SyslogConfigDetail
Returns: UpdateLogtailConfigResponse
Raise: LogException
-
update_machine_group
(project_name, group_detail)[source]¶ update machine group in a project Unsuccessful opertaion will cause an LogException.
Parameters: - project_name (string) – the Project name
- group_detail (MachineGroupDetail) – the machine group detail config
Returns: UpdateMachineGroupResponse
Raise: LogException
-
update_project_acl
(project_name, acl_action, acl_config)[source]¶ update acl of a project Unsuccessful opertaion will cause an LogException.
Parameters: - project_name (string) – the Project name
- acl_action (string) – “grant” or “revoke”, grant or revoke the acl_config to/from a project
- acl_config (acl_config.AclConfig) – the detail acl config info
Returns: UpdateAclResponse
Raise: LogException
-
update_savedsearch
(project, detail)¶ Update Savedsearch. Unsuccessful opertaion will cause an LogException.
Parameters: - project (string) – project name
- detail (dict/string) – json string
Returns: UpdateEntityResponse
Raise: LogException
-
update_shipper
(project_name, logstore_name, shipper_name, shipper_type, shipper_config)[source]¶ update odps/oss shipper for every type, it only allowed one shipper Unsuccessful opertaion will cause an LogException.
Parameters: - project_name (string) – the Project name
- logstore_name (string) – the logstore name
- shipper_name (string) – the shipper name
- shipper_type (string) – only support “odps” or “oss” , the type must be same with the oringal shipper
- shipper_config (OssShipperConfig or OdpsShipperConfig) – the detail shipper config, must be OssShipperConfig or OdpsShipperConfig type
Returns: UpdateShipperResponse
Raise: LogException
-
class
aliyun.log.
LogException
(errorCode, errorMessage, requestId='', resp_status=200, resp_header='', resp_body='')[source]¶ The Exception of the log request & response.
Parameters: - errorCode (string) – log service error code
- errorMessage (string) – detailed information for the exception
- requestId (string) – the request id of the response, ‘’ is set if client error
-
class
aliyun.log.
GetHistogramsRequest
(project=None, logstore=None, fromTime=None, toTime=None, topic=None, query=None)[source]¶ The request used to get histograms of a query from log.
Parameters: - project (string) – project name
- logstore (string) – logstore name
- fromTime (int/string) – the begin time or format of time in readable time like “%Y-%m-%d %H:%M:%S CST” e.g. “2018-01-02 12:12:10”, also support human readable string, e.g. “1 hour ago”, “now”, “yesterday 0:0:0”, refer to https://aliyun-log-cli.readthedocs.io/en/latest/tutorials/tutorial_human_readable_datetime.html
- toTime (int/string) – the end time or format of time in readable time like “%Y-%m-%d %H:%M:%S CST” e.g. “2018-01-02 12:12:10”, also support human readable string, e.g. “1 hour ago”, “now”, “yesterday 0:0:0”, refer to https://aliyun-log-cli.readthedocs.io/en/latest/tutorials/tutorial_human_readable_datetime.html
- topic (string) – topic name of logs
- query (string) – user defined query
-
class
aliyun.log.
GetLogsRequest
(project=None, logstore=None, fromTime=None, toTime=None, topic=None, query=None, line=100, offset=0, reverse=False)[source]¶ The request used to get logs by a query from log.
Parameters: - project (string) – project name
- logstore (string) – logstore name
- fromTime (int/string) – the begin time, or format of time in format “%Y-%m-%d %H:%M:%S” e.g. “2018-01-02 12:12:10”
- toTime (int/string) – the end time, or format of time in format “%Y-%m-%d %H:%M:%S” e.g. “2018-01-02 12:12:10”
- topic (string) – topic name of logs
- query (string) – user defined query
- line (int) – max line number of return logs
- offset (int) – line offset of return logs
- reverse (bool) – if reverse is set to true, the query will return the latest logs first
-
class
aliyun.log.
GetProjectLogsRequest
(project=None, query=None)[source]¶ The request used to get logs by a query from log cross multiple logstores.
Parameters: - project (string) – project name
- query (string) – user defined query
-
class
aliyun.log.
IndexConfig
(ttl=1, line_config=None, key_config_list=None, all_keys_config=None, log_reduce=None)[source]¶ The index config of a logstore
Parameters: - ttl (int) – this parameter is deprecated, the ttl is same as logstore’s ttl
- line_config (IndexLineConfig) – the index config of the whole log line
- key_config_list (dict) – dict (string => IndexKeyConfig), the index key configs of the keys
- all_keys_config (IndexKeyConfig) – the key config of all keys, the new create logstore should never user this param, it only used to compatible with old config
- log_reduce (bool) – if to enable logreduce
-
class
aliyun.log.
ListTopicsRequest
(project=None, logstore=None, token=None, line=None)[source]¶ The request used to get topics of a query from log.
Parameters: - project (string) – project name
- logstore (string) – logstore name
- token (string) – the start token to list topics
- line (int) – max topic counts to return
-
class
aliyun.log.
ListLogstoresRequest
(project=None)[source]¶ The request used to list log store from log.
Parameters: project (string) – project name
-
class
aliyun.log.
PluginConfigDetail
(logstoreName, configName, plugin, **extended_items)[source]¶ The logtail config for simple mode
Parameters: - logstoreName (string) – the logstore name
- configName (string) – the config name
- logPath (string) – folder of log path /apsara/nuwa/
- filePattern (string) – file path, e.g. .log, it will be /apsara/nuwa/…/.log
- localStorage (bool) – if use local cache 1GB when logtail is offline. default is True.
- enableRawLog (bool) – if upload raw data in content, default is False
- topicFormat (string) – “none”, “group_topic” or regex to extract value from file path e.g. “/test/(w+).log” will extract each file as topic, default is “none”
- fileEncoding (string) – “utf8” or “gbk” so far
- maxDepth (int) – max depth of folder to scan, by default its 100, 0 means just scan the root folder
- preserve (bool) – if preserve time-out, by default is False, 30 min time-out if set it as True
- preserveDepth (int) – time-out folder depth. 1-3
- filterKey (string list) – only keep log which match the keys. e.g. [“city”, “location”] will only scan files math the two fields
- filterRegex (string list) – matched value for filterKey, e.g. [“shanghai|beijing|nanjing”, “east”] note, it’s regex value list
- createTime (int) – timestamp of created, only useful when getting data from REST
- modifyTime (int) – timestamp of last modified time, only useful when getting data from REST
- extended_items (dict) – extended items
-
class
aliyun.log.
SeperatorFileConfigDetail
(logstoreName, configName, logPath, filePattern, logSample, separator, key, timeKey='', timeFormat=None, localStorage=None, enableRawLog=None, topicFormat=None, fileEncoding=None, maxDepth=None, preserve=None, preserveDepth=None, filterKey=None, filterRegex=None, createTime=None, modifyTime=None, **extended_items)[source]¶ The logtail config for separator mode
Parameters: - logstoreName (string) – the logstore name
- configName (string) – the config name
- logPath (string) – folder of log path /apsara/nuwa/
- filePattern (string) – file path, e.g. .log, it will be /apsara/nuwa/…/.log
- logSample (string) – log sample. e.g. shanghai|2000|east
- separator (string) – ‘ ‘ for tab, ‘ ‘ for space, ‘|’, up to 3 chars like “&&&” or “||” etc.
- key (string list) – keys to map the fields like [“city”, “population”, “location”]
- timeKey (string) – one key name in key to set the time or set it None to use system time.
- timeFormat (string) – whe timeKey is not None, set its format, refer to https://help.aliyun.com/document_detail/28980.html?spm=5176.2020520112.113.4.2243b18eHkxdNB
- localStorage (bool) – if use local cache 1GB when logtail is offline. default is True.
- enableRawLog (bool) – if upload raw data in content, default is False
- topicFormat (string) – “none”, “group_topic” or regex to extract value from file path e.g. “/test/(w+).log” will extract each file as topic, default is “none”
- fileEncoding (string) – “utf8” or “gbk” so far
- maxDepth (int) – max depth of folder to scan, by default its 100, 0 means just scan the root folder
- preserve (bool) – if preserve time-out, by default is False, 30 min time-out if set it as True
- preserveDepth (int) – time-out folder depth. 1-3
- filterKey (string list) – only keep log which match the keys. e.g. [“city”, “location”] will only scan files math the two fields
- filterRegex (string list) – matched value for filterKey, e.g. [“shanghai|beijing|nanjing”, “east”] note, it’s regex value list
- createTime (int) – timestamp of created, only useful when getting data from REST
- modifyTime (int) – timestamp of last modified time, only useful when getting data from REST
- extended_items (dict) – extended items
-
class
aliyun.log.
SimpleFileConfigDetail
(logstoreName, configName, logPath, filePattern, localStorage=None, enableRawLog=None, topicFormat=None, fileEncoding=None, maxDepth=None, preserve=None, preserveDepth=None, filterKey=None, filterRegex=None, **extended_items)[source]¶ The logtail config for simple mode
Parameters: - logstoreName (string) – the logstore name
- configName (string) – the config name
- logPath (string) – folder of log path /apsara/nuwa/
- filePattern (string) – file path, e.g. .log, it will be /apsara/nuwa/…/.log
- localStorage (bool) – if use local cache 1GB when logtail is offline. default is True.
- enableRawLog (bool) – if upload raw data in content, default is False
- topicFormat (string) – “none”, “group_topic” or regex to extract value from file path e.g. “/test/(w+).log” will extract each file as topic, default is “none”
- fileEncoding (string) – “utf8” or “gbk” so far
- maxDepth (int) – max depth of folder to scan, by default its 100, 0 means just scan the root folder
- preserve (bool) – if preserve time-out, by default is False, 30 min time-out if set it as True
- preserveDepth (int) – time-out folder depth. 1-3
- filterKey (string list) – only keep log which match the keys. e.g. [“city”, “location”] will only scan files math the two fields
- filterRegex (string list) – matched value for filterKey, e.g. [“shanghai|beijing|nanjing”, “east”] note, it’s regex value list
- createTime (int) – timestamp of created, only useful when getting data from REST
- modifyTime (int) – timestamp of last modified time, only useful when getting data from REST
- extended_items (dict) – extended items
-
class
aliyun.log.
FullRegFileConfigDetail
(logstoreName, configName, logPath, filePattern, logSample, logBeginRegex=None, regex=None, key=None, timeFormat=None, localStorage=None, enableRawLog=None, topicFormat=None, fileEncoding=None, maxDepth=None, preserve=None, preserveDepth=None, filterKey=None, filterRegex=None, **extended_items)[source]¶ The logtail config for full regex mode
Parameters: - logstoreName (string) – the logstore name
- configName (string) – the config name
- logPath (string) – folder of log path /apsara/nuwa/
- filePattern (string) – file path, e.g. .log, it will be /apsara/nuwa/…/.log
- logSample (string) – log sample. e.g. shanghai|2000|east
- logBeginRegex (string) – regex to match line, None means ‘.*’, just single line mode.
- regex (string) – regex to extract fields form log. None means (.*), just capture whole line
- key (string list) – keys to map the fields like [“city”, “population”, “location”]. None means [“content”]
- timeFormat (string) – whe timeKey is not None, set its format, refer to https://help.aliyun.com/document_detail/28980.html?spm=5176.2020520112.113.4.2243b18eHkxdNB
- localStorage (bool) – if use local cache 1GB when logtail is offline. default is True.
- enableRawLog (bool) – if upload raw data in content, default is False
- topicFormat (string) – “none”, “group_topic” or regex to extract value from file path e.g. “/test/(w+).log” will extract each file as topic, default is “none”
- fileEncoding (string) – “utf8” or “gbk” so far
- maxDepth (int) – max depth of folder to scan, by default its 100, 0 means just scan the root folder
- preserve (bool) – if preserve time-out, by default is False, 30 min time-out if set it as True
- preserveDepth (int) – time-out folder depth. 1-3
- filterKey (string list) – only keep log which match the keys. e.g. [“city”, “location”] will only scan files math the two fields
- filterRegex (string list) – matched value for filterKey, e.g. [“shanghai|beijing|nanjing”, “east”] note, it’s regex value list
- createTime (int) – timestamp of created, only useful when getting data from REST
- modifyTime (int) – timestamp of last modified time, only useful when getting data from REST
- extended_items (dict) – extended items
-
class
aliyun.log.
JsonFileConfigDetail
(logstoreName, configName, logPath, filePattern, timeKey='', timeFormat=None, localStorage=None, enableRawLog=None, topicFormat=None, fileEncoding=None, maxDepth=None, preserve=None, preserveDepth=None, filterKey=None, filterRegex=None, createTime=None, modifyTime=None, **extended_items)[source]¶ The logtail config for json mode
Parameters: - logstoreName (string) – the logstore name
- configName (string) – the config name
- logPath (string) – folder of log path /apsara/nuwa/
- filePattern (string) – file path, e.g. .log, it will be /apsara/nuwa/…/.log
- timeKey (string) – one key name in key to set the time or set it None to use system time.
- timeFormat (string) – whe timeKey is not None, set its format, refer to https://help.aliyun.com/document_detail/28980.html?spm=5176.2020520112.113.4.2243b18eHkxdNB
- localStorage (bool) – if use local cache 1GB when logtail is offline. default is True.
- enableRawLog (bool) – if upload raw data in content, default is False
- topicFormat (string) – “none”, “group_topic” or regex to extract value from file path e.g. “/test/(w+).log” will extract each file as topic, default is “none”
- fileEncoding (string) – “utf8” or “gbk” so far
- maxDepth (int) – max depth of folder to scan, by default its 100, 0 means just scan the root folder
- preserve (bool) – if preserve time-out, by default is False, 30 min time-out if set it as True
- preserveDepth (int) – time-out folder depth. 1-3
- filterKey (string list) – only keep log which match the keys. e.g. [“city”, “location”] will only scan files math the two fields
- filterRegex (string list) – matched value for filterKey, e.g. [“shanghai|beijing|nanjing”, “east”] note, it’s regex value list
- createTime (int) – timestamp of created, only useful when getting data from REST
- modifyTime (int) – timestamp of last modified time, only useful when getting data from REST
- extended_items (dict) – extended items
-
class
aliyun.log.
ApsaraFileConfigDetail
(logstoreName, configName, logPath, filePattern, logBeginRegex, localStorage=None, enableRawLog=None, topicFormat=None, fileEncoding=None, maxDepth=None, preserve=None, preserveDepth=None, filterKey=None, filterRegex=None, createTime=None, modifyTime=None, **extended_items)[source]¶ The logtail config for Apsara mode
Parameters: - logstoreName (string) – the logstore name
- configName (string) – the config name
- logPath (string) – folder of log path /apsara/nuwa/
- filePattern (string) – file path, e.g. .log, it will be /apsara/nuwa/…/.log
- logBeginRegex (string) – regex to match line, None means ‘.*’, just single line mode.
- localStorage (bool) – if use local cache 1GB when logtail is offline. default is True.
- enableRawLog (bool) – if upload raw data in content, default is False
- topicFormat (string) – “none”, “group_topic” or regex to extract value from file path e.g. “/test/(w+).log” will extract each file as topic, default is “none”
- fileEncoding (string) – “utf8” or “gbk” so far
- maxDepth (int) – max depth of folder to scan, by default its 100, 0 means just scan the root folder
- preserve (bool) – if preserve time-out, by default is False, 30 min time-out if set it as True
- preserveDepth (int) – time-out folder depth. 1-3
- filterKey (string list) – only keep log which match the keys. e.g. [“city”, “location”] will only scan files math the two fields
- filterRegex (string list) – matched value for filterKey, e.g. [“shanghai|beijing|nanjing”, “east”] note, it’s regex value list
- createTime (int) – timestamp of created, only useful when getting data from REST
- modifyTime (int) – timestamp of last modified time, only useful when getting data from REST
- extended_items (dict) – extended items
-
class
aliyun.log.
SyslogConfigDetail
(logstoreName, configName, tag, localStorage=None, createTime=None, modifyTime=None, **extended_items)[source]¶ The logtail config for syslog mode
Parameters: - logstoreName (string) – the logstore name
- configName (string) – the config name
- tag (string) – tag for the log captured
- localStorage (bool) – if use local cache 1GB when logtail is offline. default is True.
- createTime (int) – timestamp of created, only useful when getting data from REST
- modifyTime (int) – timestamp of last modified time, only useful when getting data from REST
- extended_items (dict) – extended items
-
class
aliyun.log.
MachineGroupDetail
(group_name=None, machine_type=None, machine_list=None, group_type='', group_attribute=None)[source]¶ The machine group detail info
Parameters: - group_name (string) – group name
- machine_type (string) – “ip” or “userdefined”
- machine_list (string list) – the list of machine ips or machine userdefined, e.g [“127.0.0.1”, “127.0.0.2”]
- group_type (string) – the machine group type, “” or “Armory”
- group_attribute (dict) – the attributes in group, it contains two optional key : 1. “externalName”: only used if the group_type is “Armory”, its the Armory name 2. “groupTopic”: group topic value
-
class
aliyun.log.
PutLogsRequest
(project=None, logstore=None, topic=None, source=None, logitems=None, hashKey=None, compress=True, logtags=None)[source]¶ The request used to send data to log.
Parameters: - project (string) – project name
- logstore (string) – logstore name
- topic (string) – topic name
- source (string) – source of the logs
- logitems (list<LogItem>) – log data
- hashKey (String) – put data with set hash, the data will be send to shard whose range contains the hashKey
- compress (bool) – if need to compress the logs
- logtags (list) – list of key:value tag pair , [(tag_key_1,tag_value_1) , (tag_key_2,tag_value_2)]
-
class
aliyun.log.
OssShipperConfig
(oss_bucket, oss_prefix, oss_role_arn, buffer_interval=300, buffer_mb=128, compress_type='snappy')[source]¶ A oss ship config
Parameters: - oss_bucket (string) – the oss bucket name
- oss_prefix (string) – the the prefix path where to save the log
- oss_role_arn (string) – the ram arn used to get the temporary write permission to the oss bucket
- buffer_interval (int) – the time(seconds) to buffer before save to oss
- buffer_mb (int) – the data size(MB) to buffer before save to oss
- compress_type (string) – the compress type, only support ‘snappy’ or ‘none’
-
class
aliyun.log.
OdpsShipperConfig
(odps_endpoint, odps_project, odps_table, log_fields_list, partition_column, partition_time_format, bufferInterval=1800)[source]¶ Odps shipper config
Parameters: - odps_endpoint (string) – the odps endpoint
- odps_project (string) – the odps project name
- odps_table (string) – the odps table name
- log_fields_list (string array) – the log field(keys in log) list mapping to the odps table column. e.g log_fields_list=[‘__time__’, ‘key_a’, ‘key_b’], the $log_time, $log_key_a, $log_key_b will mapping to odps table column No.1, No.2, No.3
- partition_column (string array) – the log fields mapping to odps table partition column
- partition_time_format (string) – the time format of __partition_time__, e.g yyyy_MM_dd_HH_mm
-
class
aliyun.log.
ShipperTask
(task_id, task_status, task_message, task_create_time, task_last_data_receive_time, task_finish_time)[source]¶ A shipper task
Parameters: - task_id (string) – the task id
- task_status (string) – one of [‘success’, ‘running’, ‘fail’]
- task_message (string) – the error message of task_status is ‘fail’
- task_create_time (int) – the task create time (timestamp from 1970.1.1)
- task_last_data_receive_time (int) – last log data receive time (timestamp)
- task_finish_time (int) – the task finish time (timestamp)
-
class
aliyun.log.es_migration.
MigrationManager
(hosts=None, indexes=None, query=None, scroll='5m', endpoint=None, project_name=None, access_key_id=None, access_key=None, logstore_index_mappings=None, pool_size=10, time_reference=None, source=None, topic=None, wait_time_in_secs=60, auto_creation=True)[source]¶ MigrationManager, migrate data from elasticsearch to aliyun log service
Parameters: - hosts (string) – a comma-separated list of source ES nodes. e.g. “localhost:9200,other_host:9200”
- indexes (string) – a comma-separated list of source index names. e.g. “index1,index2”
- query (string) – used to filter docs, so that you can specify the docs you want to migrate. e.g. ‘{“query”: {“match”: {“title”: “python”}}}’
- scroll (string) – specify how long a consistent view of the index should be maintained for scrolled search. e.g. “5m”
- endpoint (string) – specify the endpoint of your log services. e.g. “cn-beijing.log.aliyuncs.com”
- project_name (string) – specify the project_name of your log services. e.g. “your_project”
- access_key_id (string) – specify the access_key_id of your account.
- access_key (string) – specify the access_key of your account.
- logstore_index_mappings (string) – specify the mappings of log service logstore and ES index. e.g. ‘{“logstore1”: “my_index*”, “logstore2”: “index1,index2”}, “logstore3”: “index3”}’
- pool_size (int) – specify the size of process pool. e.g. 10
- time_reference (string) – specify what ES doc’s field to use as log’s time field. e.g. “field1”
- source (string) – specify the value of log’s source field. e.g. “your_source”
- topic (string) – specify the value of log’s topic field. e.g. “your_topic”
- wait_time_in_secs (int) – specify the waiting time between initialize aliyun log and executing data migration task. e.g. 60
- auto_creation (bool) – specify whether to let the tool create logstore and index automatically for you. e.g. True
\ Sort by:\ best rated\ newest\ oldest\
\\
Add a comment\ (markup):
\``code``
, \ code blocks:::
and an indented block after blank line