labeldrop doesn't seem to drop all matching labels

[leshik]

I have the following simple relabel config:

[
  {
    "source_labels": ["__aws_cloudwatch_log_group"],
    "target_label": "cloudwatch_log_group",
    "action": "replace"
  },
  {
    "regex": "__aws_.*",
    "action": "labeldrop"
  }
]

Labels before applying the config:

__aws_cloudwatch_log_group: /aws/transfer/s-7a8a3980081f4479b
__aws_cloudwatch_owner: XXXXXXXXXXXX
__aws_kinesis_event_source_arn: arn:aws:kinesis:ap-southeast-2:XXXXXXXXXXXX:stream/cloudwatch.logs
__aws_log_type: kinesis

Labels after applying the config:

__aws_cloudwatch_log_group: /aws/transfer/s-7a8a3980081f4479b
__aws_cloudwatch_owner: XXXXXXXXXXXX
__aws_kinesis_event_source_arn: arn:aws:kinesis:ap-southeast-2:XXXXXXXXXXXX:stream/cloudwatch.logs
cloudwatch_log_group: /aws/transfer/s-7a8a3980081f4479b

So the label was replaced correctly, but then only one out of 4 matching labels was dropped.

[nooop3]

Same issue here.
I found the root cause is the missing labels sorting before Prometheus relabel.Process.

This PR also fixed this issue: https://github.com/grafana/lambda-promtail/pull/98/changes#diff-bb766140b8a4690a4ee7a9dd07347619ad752e4327c5beb8e3aca182d5bf9f01R198