AWSFireLens轻松实现容器日志处理

applog应用程序和fluent-bit共享磁盘，日志内容是json格式数据，输出到S3也是JSON格式

applog应用部分在applog目录：

Dockerfile文件内容
FROM alpine
RUN mkdir -p /data/logs/
COPY testlog.sh /bin/
RUN chmod 777 /bin/testlog.sh
ENTRYPOINT ["/bin/testlog.sh"]

testlog.sh文件内容
#!/bin/sh
while :
do
   echo "{\"server_date\":\"2020-01-19\",\"hostname\":\"ip-172-31-43-24.cn-northwest-1.compute.internal\",\"pid\":5404,\"method\":\"POST\",\"clientIP\":\"10.11.12.13\",\"countryCode\":\"ID\",\"url\":\"/v1/mail/list\",\"status\":\"200\",\"latency\":7,\"length\":24,\"userId\":9536605,\"code\":20001}" >> /data/logs/access.log
   echo "{\"server_date\":\"2020-01-19\",\"hostname\":\"ip-172-31-43-24.cn-northwest-1.compute.internal\",\"pid\":1000,\"method\":\"GET\",\"clientIP\":\"20.21.22.23\",\"countryCode\":\"ID\",\"url\":\"/v1/mail/list\",\"status\":\"500\",\"latency\":10,\"length\":12,\"userId\":1010001,\"code\":10001}" >> /data/logs/error.log
sleep 10
done

fluent-bit部分在fluent-bit目录：

Dockerfile文件内容
FROM amazon/aws-for-fluent-bit:latest
ADD extra.conf /extra.conf

extra.conf文件内容
[SERVICE]
    Parsers_File /fluent-bit/parsers/parsers.conf
    Flush 1
    Grace 30

[INPUT]
    Name tail
    Path /data/logs/access.log
    Tag access

[INPUT]
    Name tail
    Path /data/logs/error.log
    Tag error

[FILTER]
    Name parser
    Match *
    Key_Name log
    Parser json  
    Reserve_Data True

[OUTPUT]
    Name firehose
    Match access
    region us-east-1
    delivery_stream fluentbit-access

[OUTPUT]
    Name firehose
    Match error
    region us-east-1
    delivery_stream fluentbit-error  #要在firehouse里面设置
    
----------------
报错信息/ecs/firelens-sample/文件夹下面ecs/log_router/03aafe7fa1f4452d862854b33311190f

打镜像步骤：
cd fleuntbit
docker build -t fleuntbit:v3 .
docker tag fleuntbit:v3 402097323/fleuntbit:v3
docker push 402097323/fleuntbit:v3

dockerup账号******/******

taskdef.json:
{
    "family": "firelens-sample",
    "taskRoleArn": "arn:aws:iam::254278701124:role/ecsTaskExecutionRole",
    "executionRoleArn": "arn:aws:iam::254278701124:role/ecsTaskExecutionRole",
    "containerDefinitions": [
        {
            "essential": true,
            "name": "log_router",
            "image": "402097323/fleuntbit:v3",
            "logConfiguration": {
                "logDriver": "awslogs",
                "options": {
                    "awslogs-create-group": "true",
                    "awslogs-group": "/ecs/firelens-sample",
                    "awslogs-region": "us-east-1",
                    "awslogs-stream-prefix": "ecs"
                }
            },
            "mountPoints": [
                {
                    "sourceVolume": "data",
                    "containerPath": "/data",
                    "readOnly": false
                }
            ],
            "firelensConfiguration": {
                "type": "fluentbit",
                "options": {
                    "config-file-type": "file",
                    "config-file-value": "/extra.conf",
                    "enable-ecs-log-metadata": "false"  #如果不设置enable-ecs-log-metadata 为 false，则日志条目中会包括以下元数据信息：1、ecs_cluster任务所属的集群的名称；2、ecs_task_arn容器所属的任务的完整ARN；3、ecs_task_definition任务正在使用的任务定义名称和修订；4、ec2_instance_id容器托管于的 Amazon EC2实例ID。此字段仅对使用EC2启动类型的任务有效
                }
            }
            "user": "0",
        },
        {
            "essential": true,
            "name": "myapp",
            "image": "402097323/applog:v2",
            "logConfiguration": {
                "logDriver": "awsfirelens"
            }
            "mountPoints": [
                {
                    "sourceVolume": "data",
                    "containerPath": "/data",
                    "readOnly": false
                }
            ],
            "dependsOn": [
                {
                    "containerName": "log_router",
                    "condition": "START"
                }
            ],

        }
    ],
    "cpu": "256",
    "memory": "512"
    "volumes": [
        {
            "name": "data",
            "host": {}
        }
    ],
    "compatibilities": [
        "EC2",
        "FARGATE"
    ],
    "requiresCompatibilities": [
        "FARGATE"
    ],
     "networkMode": "awsvpc"
}

创建 Amazon Kinesis Firehose 传输流和 S3 存储桶，S3对应存储桶如图，要设置S3存储桶权限为允许公开,ref:  使用 AWS FireLens 轻松实现 AWS Fargate 容器日志处理 | 亚马逊AWS官方博客