Description
Jikkou v0.37.3 fails to authenticate to AWS MSK via IAM when running on an EKS pod using EKS Pod Identity for credential injection.
Environment
- Jikkou: v0.37.3 (Linux x86_64 binary)
- MSK: Express brokers, KRaft, IAM auth on port 9098
- EKS: v1.33, Pod Identity agent running
- Runner: GitHub Actions Runner Controller (ARC) on EKS
Symptoms
SaslAuthenticationException: Access denied
on all three brokers, repeatedly until timeout:
ERROR o.apache.kafka.clients.NetworkClient - Connection to node -1 failed authentication due to: Access denied
ERROR i.s.j.k.internals.KafkaBrokersReady - Error while listing Kafka nodes: SaslAuthenticationException: Access denied
JikkouRuntimeException: Timeout expired. The timeout period elapsed prior to the requested number of kafka brokers is available.
Pod Identity credentials are valid
From the same container, AWS CLI and Python boto3 successfully use the Pod Identity credentials:
$ aws sts get-caller-identity
{
"Account": "<account_id>",
"Arn": "arn:aws:sts::<account_id>:assumed-role/<role-name>/eks-..."
}The IAM role has kafka-cluster:Connect, kafka-cluster:DescribeCluster, and other kafka-cluster:* permissions on the MSK cluster.
Pod Identity env vars
EKS Pod Identity injects credentials via:
AWS_CONTAINER_CREDENTIALS_FULL_URI=http://169.254.170.23/v1/credentials
AWS_CONTAINER_AUTHORIZATION_TOKEN_FILE=/var/run/secrets/pods.eks.amazonaws.com/serviceaccount/eks-pod-identity-token
AWS_DEFAULT_REGION=us-east-1
AWS_REGION=us-east-1
AWS_STS_REGIONAL_ENDPOINTS=regional
The credential endpoint returns valid temporary credentials when called with the token.
Jikkou config
jikkou {
kafka {
client {
bootstrap.servers = "<broker1>:9098,<broker2>:9098,<broker3>:9098"
security.protocol = SASL_SSL
sasl.mechanism = AWS_MSK_IAM
sasl.jaas.config = "software.amazon.msk.auth.iam.IAMLoginModule required; "
sasl.client.callback.handler.class = software.amazon.msk.auth.iam.IAMClientCallbackHandler
}
}
}Workaround
IRSA (IAM Roles for Service Accounts) works. The issue is specific to Pod Identity's AWS_CONTAINER_AUTHORIZATION_TOKEN_FILE-based credential chain.
Suspected cause
The aws-msk-iam-auth library (or the underlying AWS SDK) may not fully support the token-file-based container credential provider used by EKS Pod Identity. IRSA uses a different mechanism (OIDC projected service account tokens via AWS_WEB_IDENTITY_TOKEN_FILE) which is supported.
Description
Jikkou v0.37.3 fails to authenticate to AWS MSK via IAM when running on an EKS pod using EKS Pod Identity for credential injection.
Environment
Symptoms
on all three brokers, repeatedly until timeout:
Pod Identity credentials are valid
From the same container, AWS CLI and Python boto3 successfully use the Pod Identity credentials:
$ aws sts get-caller-identity { "Account": "<account_id>", "Arn": "arn:aws:sts::<account_id>:assumed-role/<role-name>/eks-..." }The IAM role has
kafka-cluster:Connect,kafka-cluster:DescribeCluster, and otherkafka-cluster:*permissions on the MSK cluster.Pod Identity env vars
EKS Pod Identity injects credentials via:
The credential endpoint returns valid temporary credentials when called with the token.
Jikkou config
Workaround
IRSA (IAM Roles for Service Accounts) works. The issue is specific to Pod Identity's
AWS_CONTAINER_AUTHORIZATION_TOKEN_FILE-based credential chain.Suspected cause
The
aws-msk-iam-authlibrary (or the underlying AWS SDK) may not fully support the token-file-based container credential provider used by EKS Pod Identity. IRSA uses a different mechanism (OIDC projected service account tokens viaAWS_WEB_IDENTITY_TOKEN_FILE) which is supported.