Problem

In our recent work, we encountered a problem where we changed the secret in the vault and updated the way to retrieve it. This change didn’t cause any issues during local testing, but problems arose after deploying to staging. We hope to check what is being used in remote environments so that we can determine whether there’s an issue with our retrieval method or if there’s a problem with the secret itself.

This kind of issue is common in automated deployment environments because stg/prd usually involves a series of automatic injections and startup scripts. Therefore, this method helps us check conflicts between local and remote automated deployments.

Debug

Env Setup

First, we create a demo service with a single endpoint and an injected value.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18

package com.example.demo;

import org.springframework.beans.factory.annotation.Value;
import org.springframework.web.bind.annotation.GetMapping;
import org.springframework.web.bind.annotation.RestController;

@RestController
public class DemoService {

    @Value("${demo.value}")
    private String stringValue;

    @GetMapping("/demo")
    public String getStringValue() {
        return stringValue;
    }

}

Next, we input the value into the application.yaml config. This value can be overridden by an injected environment variable.

1
2

demo:
  value: ${DEMO_VALUE:123}

Finally, we can construct the Docker image and run it to mimic the service deployment in a remote environment.

1
2

CONTAINER ID   IMAGE     COMMAND               CREATED         STATUS         PORTS                    NAMES
c64090609d8a   demo      "java -jar app.jar"   8 seconds ago   Up 7 seconds   0.0.0.0:8080->8080/tcp   charming_chatterjee

Install Arthas

Arthas allows developers to troubleshoot production issues for Java applications without modifying code or restarting servers.

1
2
3
4
5
6
7

# log into container
docker exec -ti c64090609d8a bash

# install and start arthas
bash-4.4# curl -O https://arthas.aliyun.com/arthas-boot.jar

bash-4.4# java -jar arthas-boot.jar

Check the target service

We need to locate the target service bean within the Spring Context. A simple method is to monitor the Spring bean, so we do not need to manually create a custom bean with ApplicationContextAware.

1

tt -t org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter invokeHandlerMethod

Arthas assists in marking the target, allowing us to retrieve the application context and examine the bean as needed. The index from the previous step is 1000.

1

tt -i 1000 -w 'target.getApplicationContext().getBean("demoService")'

We’ve noted that the injected value is 123, showing we can retrieve this value from the remote runtime.

In our case, the service’s injected value didn’t meet our expectations. So we traced the problem to a conflict in retrieving this value from the automated deployment pipeline and have since resolved it.

More Usecase

The above workflow has more applications. For instance, we can invoke the method in the bean to verify if the outcome is as expected.

1
2
3

[arthas@1]$ tt -i 1000 -w 'target.getApplicationContext().getBean("demoService").getStringValue()'
@String[123]
#ffect(row-cnt:1) cost in 1 ms.

Or we can use ognl to check more info for static methods or property. For example, we can create a custom bean and implement ApplicationContextAware interface, set the context as static property, then we can use ognl to get context directly.

1
2
3

sc -d com.xxx.SpringContexDemo

ognl -c <index> '@com.xxx.SpringContexDemo@getApplicationContext()'

Summary

This debugging workflow allows us to access the ApplicationContext in a remote environment, aiding online issue resolution. It’s not just useful for verifying injected values; it can also help resolve more complex issues since we have access to the context and all its beans.

Reference

https://github.com/alibaba/arthas

https://github.com/alibaba/arthas/issues/482