Apigee Hybrid - Workload Identity Federation to replace GCP Service account Keys

[Ashk1]

Hi @dchiesa1 ,

I am trying to help out someone with a java callout code. and when we deploy the callout proxy, the deployment seems to be successful but when the API is triggerred we see the following errors:
I am not even a newbee in Apigee related stuff so not able to debug correctly.

{"level":"SEVERE","thread":"Apigee-Main-61","mdc":{"apiName":"test-api-v1","org":"xx-apigeehyb-test-app-cbe6","messageId":"xxxxx-2271-4cbc-bc1d-xxxxxx","env":"glb-test1-3","revision":"1","trackingId":"xxxx-872e-4f6a-a32a-xxxxx"},"className":"com.apigee.flow.execution.AbstractAsyncExecutionStrategy$AsyncExecutionTask","method":"logException","severity":"SEVERE","message":"Exception caught with message: Failed to execute JavaCallout. loader constraint violation: when resolving method \u0027org.slf4j.ILoggerFactory org.slf4j.impl.StaticLoggerBinder.getLoggerFactory()\u0027 the class loader com.apigee.messaging.resource.JavaResourceClassLoader @579d033b of the current class, org/slf4j/LoggerFactory, and the class loader \u0027app\u0027 for the method\u0027s defining class, org/slf4j/impl/StaticLoggerBinder, have different Class objects for the type org/slf4j/ILoggerFactory used in the signature (org.slf4j.LoggerFactory is in unnamed module of loader com.apigee.messaging.resource.JavaResourceClassLoader @579d033b, parent loader com.apigee.messaging.resource.DeprecatingClassesLoader @303669ef; org.slf4j.impl.StaticLoggerBinder is in unnamed module of loader \u0027app\u0027)","formattedDate":"2024-03-05T18:52:12.465Z"

I also see a bunch of SEVERE message before this error, could this be an after math of permission denied error or vice versa.

More logs:

{"level":"SEVERE","thread":"Apigee-Main-61","mdc":{"apiName":"test-api-v1","org":"xx-apigeehyb-test-app-cbe6","policyName":"JavaCallout.AWS.STSToken","messageId":"xxxx-2271-4cbc-bc1d-xxx","env":"glb-xxxx-3","revision":"1","trackingId":"xxxx-872e-4f6a-a32a-xxxx"},"className":"com.apigee.securitypolicy.SecurityPolicy$3","method":"run","severity":"SEVERE","message":"Permission denied (\"java.io.FilePermission\" \"/application/opt/apigee/apigee-runtime/webapps/api/org/slf4j/impl/StaticLoggerBinder.class\" \"read\") ProtectionDomain\u003dProtectionDomain (file:/application/opt/apigee/apigee-runtime/data/work/contract_glb-xxxx-3_264/a/shared-flow-v1__38/java/awsclient-1.0-SNAPSHOT.jar/ \u003cno signer certificates\u003e)\n JavaResourceClassLoader /application/opt/apigee/apigee-runtime/data/work/contract_glb-xxxx-3_264/a/shared-flow-v1__38/java\n \u003cno principals\u003e\n java.security.Permissions@36556a8d (\n (\"com.apigee.securitypolicy.RestrictedSocketPermission\" \"RestrictedSocketPermission\")\n (\"java.io.FilePermission\" \"/usr/lib/jvm/java-11-openjdk-amd64/-\" \"read,readlink\")\n (\"java.io.FilePermission\" \"/application/opt/apigee/apigee-runtime/data/work/contract_glb-xxxx-3_264/a/shared-flow-v1__38/java/../../../java/-\" \"read,readlink\")\n (\"java.io.FilePermission\" \"/tmpfs/src/gfile/jre/-\" \"read,readlink\")\n (\"java.io.FilePermission\" \"${javacallout.org.dir}/-\" \"read\")\n (\"java.io.FilePermission\" \"/application/opt/apigee/apigee-runtime/data/work/contract_glb-xxxx-3_264/a/shared-flow-v1__38/java/../../../../../java/-\" \"read,readlink\")\n (\"java.io.FilePermission\" \"/application/opt/apigee/apigee-runtime/data/work/contract_glb-xxxx-3_264/a/shared-flow-v1__38/java/awsclient-1.0-SNAPSHOT.jar/-\" \"read\")\n (\"java.io.FilePermission\" \"/application/opt/apigee/apigee-runtime/data/work/contract_glb-xxxx-3_264/a/shared-flow-v1__38/java/-\" \"read,readlink\")\n (\"java.lang.RuntimePermission\" \"accessClassInPackage.*\")\n (\"java.lang.RuntimePermission\" \"modifyThreadGroup\")\n (\"java.lang.RuntimePermission\" \"getProtectionDomain\")\n (\"java.lang.RuntimePermission\" \"getClassLoader\")\n (\"java.lang.RuntimePermission\" \"modifyThread\")\n (\"java.lang.RuntimePermission\" \"enableContextClassLoaderOverride\")\n (\"java.lang.RuntimePermission\" \"setContextClassLoader\")\n (\"java.lang.RuntimePermission\" \"stopThread\")\n (\"java.net.NetPermission\" \"specifyStreamHandler\")\n (\"java.util.PropertyPermission\" \"*\" \"read\")\n (\"java.net.URLPermission\" \"ftp:*\" \"*:\")\n (\"java.net.URLPermission\" \"https:*\" \"*:\")\n (\"java.net.URLPermission\" \"sftp:*\" \"*:\")\n (\"java.net.URLPermission\" \"http:*\" \"*:\")\n (\"com.apigee.securitypolicy.AllDenyAndLogExcept\" \"Alldenyandlogexcept\" \"java.io.FilePermission,java.net.SocketPermission,java.util.PropertyPermission,java.net.URLPermission,java.net.NetPermission:specifyStreamHandler,java.lang.RuntimePermission:modifyThread/stopThread/modifyThreadGroup/getProtectionDomain/getClassLoader/setContextClassLoader/enableContextClassLoaderOverride,java.io.SerializablePermission:enableSubclassImplementation\")\n (\"java.io.SerializablePermission\" \"enableSubclassImplementation\")\n)\n\n Grant\u003djava.security.Permissions@4e48096c (\n (\"com.apigee.securitypolicy.RestrictedSocketPermission\" \"RestrictedSocketPermission\")\n (\"java.io.FilePermission\" \"/usr/lib/jvm/java-11-openjdk-amd64/-\" \"read,readlink\")\n (\"java.io.FilePermission\" \"/application/opt/apigee/apigee-runtime/data/work/contract_glb-xxxx-3_264/a/shared-flow-v1__38/java/../../../java/-\" \"read,readlink\")\n (\"java.io.FilePermission\" \"/tmpfs/src/gfile/jre/-\" \"read,readlink\")\n (\"java.io.FilePermission\" \"${javacallout.org.dir}/-\" \"read\")\n (\"java.io.FilePermission\" \"/application/opt/apigee/apigee-runtime/data/work/contract_glb-xxxx-3_264/a/shared-flow-v1__38/java/../../../../../java/-\" \"read,readlink\")\n (\"java.io.FilePermission\" \"/application/opt/apigee/apigee-runtime/data/work/contract_glb-xxxx-3_264/a/shared-flow-v1__38/java/-\" \"read,readlink\")\n (\"java.util.PropertyPermission\" \"*\" \"read\")\n (\"java.net.NetPermission\" \"specifyStreamHandler\")\n (\"java.lang.RuntimePermission\" \"modifyThreadGroup\")\n (\"java.lang.RuntimePermission\" \"accessClassInPackage.*\")\n (\"java.lang.RuntimePermission\" \"getProtectionDomain\")\n (\"java.lang.RuntimePermission\" \"modifyThread\")\n (\"java.lang.RuntimePermission\" \"getClassLoader\")\n (\"java.lang.RuntimePermission\" \"enableContextClassLoaderOverride\")\n (\"java.lang.RuntimePermission\" \"stopThread\")\n (\"java.lang.RuntimePermission\" \"setContextClassLoader\")\n (\"java.net.URLPermission\" \"https:*\" \"*:\")\n (\"java.net.URLPermission\" \"ftp:*\" \"*:\")\n (\"java.net.URLPermission\" \"sftp:*\" \"*:\")\n (\"java.net.URLPermission\" \"http:*\" \"*:\")\n (\"com.apigee.securitypolicy.AllDenyAndLogExcept\" \"Alldenyandlogexcept\" \"java.io.FilePermission,java.net.SocketPermission,java.util.PropertyPermission,java.net.URLPermission,java.net.NetPermission:specifyStreamHandler,java.lang.RuntimePermission:modifyThread/stopThread/modifyThreadGroup/getProtectionDomain/getClassLoader/setContextClassLoader/enableContextClassLoaderOverride,java.io.SerializablePermission:enableSubclassImplementation\")\n (\"java.io.SerializablePermission\" \"enableSubclassImplementation\")\n)\n","formattedDate":"2024-03-05T18:52:12.462Z","logger":"SERVICES.SECURITY_POLICY_SERVICE"}

{"level":"WARNING","thread":"Apigee-Main-61","mdc":{"apiName":"test-api-v1","org":"xx-apigeehyb-test-app-cbe6","policyName":"JavaCallout.AWS.STSToken","messageId":"xxxx-2271-4cbc-bc1d-xxx","env":"glb-xxxx-3","revision":"1","trackingId":"xxxx-872e-4f6a-a32a-xxxx"},"className":"com.apigee.messaging.resource.LogOnce","method":"lambda$warnLog$0","severity":"WARNING","message":"Non Apigee class loaded from Gateway ClassLoader : org.slf4j.impl.StaticLoggerBinder","formattedDate":"2024-03-05T18:52:12.464Z","logger":"CONFIG-CHANGE"}

{"level":"SEVERE","thread":"Apigee-Main-61","mdc":{"apiName":"test-api-v1","org":"xx-apigeehyb-test-app-cbe6","messageId":"xxxx-2271-4cbc-bc1d-xxx","env":"glb-xxxx-3","revision":"1","trackingId":"xxxx-872e-4f6a-a32a-xxxx"},"className":"com.apigee.flow.execution.AbstractAsyncExecutionStrategy$AsyncExecutionTask","method":"logException","severity":"SEVERE","message":"Exception caught with message: Failed to execute JavaCallout. loader constraint violation: when resolving method \u0027org.slf4j.ILoggerFactory org.slf4j.impl.StaticLoggerBinder.getLoggerFactory()\u0027 the class loader com.apigee.messaging.resource.JavaResourceClassLoader @579d033b of the current class, org/slf4j/LoggerFactory, and the class loader \u0027app\u0027 for the method\u0027s defining class, org/slf4j/impl/StaticLoggerBinder, have different Class objects for the type org/slf4j/ILoggerFactory used in the signature (org.slf4j.LoggerFactory is in unnamed module of loader com.apigee.messaging.resource.JavaResourceClassLoader @579d033b, parent loader com.apigee.messaging.resource.DeprecatingClassesLoader @303669ef; org.slf4j.impl.StaticLoggerBinder is in unnamed module of loader \u0027app\u0027)","formattedDate":"2024-03-05T18:52:12.465Z","logger":"MESSAGING.FLOW","exceptionStackTrace":"com.apigee.kernel.exceptions.spi.UncheckedException{ code \u003d steps.javacallout.ExecutionError, message \u003d Failed to execute JavaCallout. loader constraint violation: when resolving method \u0027org.slf4j.ILoggerFactory org.slf4j.impl.StaticLoggerBinder.getLoggerFactory()\u0027 the class loader com.apigee.messaging.resource.JavaResourceClassLoader @579d033b of the current class, org/slf4j/LoggerFactory, and the class loader \u0027app\u0027 for the method\u0027s defining class, org/slf4j/impl/StaticLoggerBinder, have different Class objects for the type org/slf4j/ILoggerFactory used in the signature (org.slf4j.LoggerFactory is in unnamed module of loader com.apigee.messaging.resource.JavaResourceClassLoader @579d033b, parent loader com.apigee.messaging.resource.DeprecatingClassesLoader @303669ef; org.slf4j.impl.StaticLoggerBinder is in unnamed module of loader \u0027app\u0027), associated contexts \u003d []}\n\tat com.apigee.kernel.exceptions.spi.UncheckedException.setPrintStack(UncheckedException.java:147)\n\tat com.apigee.steps.javacallout.JavaCalloutStepDefinition$SecurityWrappedExecution.execute(JavaCalloutStepDefinition.java:290)\n\tat com.apigee.steps.javacallout.JavaCalloutStepDefinition$CallOutWrapper.execute(JavaCalloutStepDefinition.java:108)\n\tat com.apigee.messaging.runtime.steps.StepExecution.execute(StepExecution.java:175)\n\tat com.apigee.flow.execution.AbstractAsyncExecutionStrategy$AsyncExecutionTask.call(AbstractAsyncExecutionStrategy.java:107)\n\tat com.apigee.flow.execution.AsyncExecutionStrategy.execute(AsyncExecutionStrategy.java:30)\n\tat com.apigee.flow.MessageFlowImpl.execute(MessageFlowImpl.java:617)\n\tat com.apigee.flow.MessageFlowImpl.resume(MessageFlowImpl.java:433)\n\tat com.apigee.flow.execution.ExecutionContextImpl.lambda$resume$0(ExecutionContextImpl.java:143)\n\tat com.apigee.threadpool.ThreadPoolManager.submitOnMainThreadPool(ThreadPoolManager.java:235)\n\tat com.apigee.flow.execution.ExecutionContextImpl.resume(ExecutionContextImpl.java:151)\n\tat com.apigee.messaging.adaptors.http.configuration.MessageProcessorHttpSkeletonFactory$MessageProcessorRequestListener.lambda$onHeaders$1(MessageProcessorHttpSkeletonFactory.java:517)\n\tat com.apigee.threadpool.RunnableWrapperForMDCPreservation.run(RunnableWrapperForMDCPreservation.java:22)\n\tat com.apigee.threadpool.QueueMetricsAwareTask.run(QueueMetricsAwareTask.java:31)\n\tat java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)\n\tat java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)\n\tat java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)\n\tat java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)\n\tat java.base/java.lang.Thread.run(Thread.java:829)\n"}

In my pom.xml I do not have any slf4j dependency, but I do see that my aws related code is dependent on it. So I am not planning to include slf4j-simple jar in pom file and try again tomorrrow.

Anything else you can suggest?
Please help.

[dchiesa1]

I suspect you are running into a permissions problem. 

This is relevant: https://www.googlecloudcommunity.com/gc/Apigee/Apigee-Edge-unable-to-initialize-IBM-MQ-client-classe...

For Apigee hybrid, you have the ability to relax the permissions for your Java callouts. 

https://cloud.google.com/apigee/docs/api-platform/develop/adding-custom-java-callout-security-policy....

Some further comments.

1. Take care in modifying permissions. It may have downstream effects that may affect the performance and stability of your Hybrid clusters.

2. I don't know what you're doing with AWS, but , you may not want to load the entire AWS SDK JAR into an Apigee callout. Despite the flexibility of the Java callout mechanism, Apigee is not intended to be a general-purpose application server. Some people want to merely connect to an AWS endpoint, and for that they want an AWS signature. There is an existing community-contributed Java callout that does this, without the use of the AWS SDK JARs, which means (a) there's no permissions problem, and (b) it's a smaller, lighter-weight JAR.  

[Ashk1]

Thank you for taking time in looking into this.

Regrading the errors first:
So after including slf4j jar as a dependency in pom file. I do not see the errors I saw yesterday. But definitely there are permissions issues as  I see this now:
"message":"Permission denied (\"java.lang.RuntimePermission\" \"getenv.AWS_PROFILE\")
"message":"Permission denied (\"java.lang.RuntimePermission\" \"getenv.AWS_SHARED_CREDENTIALS_FILE\")
"message":"Permission denied (\"java.lang.RuntimePermission\" \"getenv.HOME\")
"message":"Permission denied (\"java.io.FilePermission\" \"/opt/apigee/.aws/credentials\" \"read\")

Q: Do you think providing these permissions in java security policies would have underlying impact on hybrid cluster performance or cause any further security issues?
Something like this?
grant {
permission java.lang.RuntimePermission "getenv.AWS_PROFILE";
permission java.lang.RuntimePermission "getenv.AWS_SHARED_CREDENTIALS_FILE";
permission java.lang.RuntimePermission "getenv.HOME";
};
I am not sure how do I give permission for "message":"Permission denied (\"java.io.FilePermission\" \"/opt/apigee/.aws/credentials\" \"read\"), as there is no such file existing, we plan to use IAM role attached to the EKS POD. Not sure why the code is looking for this file even, may be due to default authentication chain? Will investigate more.

Regarding what I am trying to do:
Apigee uses service account keys (plain text json) files for Google integrations, and due to security reasons, we are trying to get rid of SA keys. Hence, I am trying to get WIF to work with Apigee Hybrid, which is on EKS cluster on AWS - The native WIF configuration with EKS does not work, as when we use service callouts the AWS metadata service is unreachable as it looks like the comms breaking out to internet and AWS metadata server is only accessible within the compute services. The other way is to use rest apis to get AWS id_token(ref: Configure workload identity federation with AWS or Azure  |  IAM Documentation  |  Google Cloud) 
So I am able to get signed headers and construct required payload for Google STS service and then exchange the STS token with Google Service account access token all in a java code.  Some more info here: Google Workload identity federation using AWS sigv4 signed request : r/googlecloud (reddit.com) 

I agree that the signing header part of this can be achieved by your Java callout that does this, without the use of the AWS SDK JARs, and its brilliant. The only issue I have with it is that it requires AWS access key and secret. And we do not want to use them since the java callout runs on Apigee Hybrid runtime on EKS, so we are trying to leverage AWS InstanceProfile (IAM role) attached to the EKS node to provide the default credentials to sign the request using AWS sigv4 algorithm. 
I did try to modify your callout to use InstanceProfile Credentials instead of AWS key and secret, but I haven't tested it yet. But I am guessing that I would end up facing the same permissions issues with this as well.
Is there anything else you can think of?

I hope this gives some background.
 

[dchiesa1]

HA, thanks for all that detail.

regarding

The native WIF configuration with EKS does not work, as when we use service callouts the AWS metadata service is unreachable as it looks like the comms breaking out to internet and AWS metadata server is only accessible within the compute services.

Reading here, it seems the intention is to allow systems on EKS to reach the AWS metadata service. I am no AWS expert, but it sounds like it should be possible, specifically with IMDSv2. Are you using the PUT call? If not, try it? If so, maybe there is one additional hop that you are seeing, by using ServiceCallout, and you are breaching the hop limit of 2? So you would need to raise it to 3?

https://aws.amazon.com/about-aws/whats-new/2020/08/amazon-eks-supports-ec2-instance-metadata-service...

regarding

Do you think providing these permissions in java security policies would have underlying impact on hybrid cluster performance or cause any further security issues?

Not necessarily. The source of my concern is that with the proposed change in permissions, ANY Java callout could reference those environment variables. Maybe not an issue for you as you already have a secure software lifecycle and you already analyze Java code that gets compiled into Apigee callouts. But what about the JARs that code pulls in as dependency? Are you analyzing those too? The log4shell attack showed that you need to consider each jar or library. It's not enough to scan your own code. So I would say it just opens one additional bit of surface area that you would need to watch.

re

\"java.io.FilePermission\" \"/opt/apigee/.aws/credentials\" \"read\"),

I have seen this kind of thing before. If it follows the same pattern, the library code is first looking for environment variables to find the location of the file to read, and finding no environment variables, falls back to try to read the "Default" path to the credentials file. So probably you would need to enable filesystem access to Something. even if not the default path.

I also found that you could set a java system property aws.sharedCredentialsFile , to override the env variable. So maybe you don't need to add environment permissions at all. But still you may need to add permissions to read properties. And of course the filesystem properties.

Good luck!

[Ashk1]

Thank you @dchiesa1 

I guess I have few options to try then:

1. Test ServiceCallout again - Set hop limit to 3 if ServiceCallout I causing this problem.
2. Test the JavaCallout again after assigning the java permission in security policy to read environment and properties.
3. Is it worth configuring one AWS secret key and access key in flow, so we can use Apigee community proxy (https://github.com/DinoChiesa/Apigee-Java-AWSV4-Signature) instead of our own custom code?  The rationale is one AWS secret key can be used to sign the requests which will help us avoid configuring individual GCP service account keys for integration with GCP services.

I stumbled upon one more thing:
The hybrid runtime already has runtime GCP service accounts created. Can we not use that runtime SA to generate a Google ID_token somehow  and use that to be exchanged with Google STS for SA access token instead of trusting AWS provided ID_token for Workload Identity federation.
This page shows how Apigee performs the token generation and makes secure calls to targeted Google services or custom hosted services for you, without the need to manually set authentication headers or otherwise modify a service request.

Can this be leveraged for the first part only. i.e. Apigee performs the token generation, and not call any endpoint. Like just set the Id_token in some variable, which can be used in next step.

[dchiesa1]

That page talks about the Authentication element that can be applied to ServiceCallout or etc. If you use that configuration element, then Apigee will, behind the scenes, make a call to iamcredentials to... request credentials to impersonate a specific service account. It eventually, through various layers of software, boils down to doing this:

$ curl -i -X POST -H "Authorization: Bearer $ACCESS_TOKEN" \
 -H "Content-Type: application/json; charset=utf-8" \
 "https://iamcredentials.googleapis.com/v1/projects/-/serviceAccounts/${SA}:generateAccessToken" \
 -d '{
  "scope" : ["https://www.googleapis.com/auth/platform"],
  "lifetime": "3599s"
 }'

...from within the Apigee runime. And the response to that is like so:

{
  "accessToken": "ya29.c.b0AXv0zTOxsOanHJbPqyGCvwG0w-e3wrsQoFlyyc0P8u6xk",
  "expireTime": "2024-03-08T23:02:45Z"
}

Basically it presents the access token for ...some principal... and gets an access token for another principal (though you can configure the Authentication element to alternatively request an ID Token). The first principal is, I suppose, the GCP Service account associated to the Apigee hybrid runtime. The second is the GCP SA that you want to impersonate.

As the documentation page that you cited explains, you can attach that Authentication element to any HTTPTargetEndpoint, or to a ServiceCallout or to an ExternalCallout (for GRPC). And Apigee will get the right token, and of course cache it for the correct amount of time. And inject that token into the Authorization header in the outbound call for the Target, the ServiceCallout or the ExternalCallout, respectively.

Hence, I am trying to get WIF to work with Apigee Hybrid, which is on EKS cluster on AWS -

I guess this means ... you want to eventually get a token for a specific GCP SA and then use it to connect to something in Google cloud? I just re-read what you wrote, and you mentioned "Google Integrations". So I guess this is right. Then yes, the Authentication element will be your friend. You don't need no stinking java. Or AWS SDK. Or WIF. You WILL have to deploy each API proxy that uses this element with a GCP Service Account. It can be different for each proxy, but for each proxy there can be only one. No need for an SA Key, just the SA Email. You need to specify the SA email at the time you deploy the proxy.

Just curious, What is the GCP endpoint you are eventually connecting to from your proxies running in Apigee hybrid?