graphql with map type

When graphql server (dgs framework) generate the graphql response, the Map attribute is rendered as {key=value}.
So for example

class DataObject{
....
Map<String, Double> cashByCurrencyUnadjustedUsd;
}

the dataFetcher returned the result as

from the client side, its also using dgs to parse the graphql response to DataObject.

Expected behavior

I would expect the client has no issue to parse the data, considering they are from the same project (dgs here).

Actual behavior

A jackson MismatchedInputException was thrown here:

Steps to reproduce

  1. Created a POJO with a map attribute
  2. use dgs data fetcher to response above pojo
  3. use dgs client to parse the graphql response into same POJO

https://github.com/Netflix/dgs-framework/issues/1149

the possible solution

Looks like this probably is an issue with graphql-java and graphql in general where Map is not supported type. I have tried to create a custom scalar type, something like

# in schema
scalar Map

# java code
@DgsScalar(name="Map")
public class DgsMap ....{
    public **String** serialize (Object dataFetcherResult) ..{
        return mapper.writeValueAsString(dataFetcherResult);
   }
}

this then would result out

which Jackson from dgs-client still has issue to parse.

In the end, I have left dgs-framework to parse the Map as String using default Map.toString(). Then from dgs-client, I swithed to Gson as a custom deserializer to parse the String back to Map.

Not sure whether that’s the optimal approach, but seems like thats the only option I found working at the moment.

so basically, there is no change needed on the dgs framework side. However, from the client, when dgs-client parse the response, it would use Gson to parse the `Map.toString` value.

something like

@JsonProperties(ignoreUnknown=true)
public class POJO{ //on client side

....

@Deserialize(CustomDeserializer.class)
Map<> cashByCurrencyUnadjustedUsd;
}


then in the CustomDeserializer, it basically deserialize the string using gson.

public class CustomDeserializerextends StdDeserializer<Map> { 

    Gson gson = new Gson();
    public CustomDeserializer() { 
        this(null); 
    } 

    public CustomDeserializer(Class<?> vc) { 
        super(vc); 
    }

    @Override
    public Map deserialize(JsonParser jp, DeserializationContext ctxt) 
      throws IOException, JsonProcessingException {
        TextNode node = jp.getCodec().readTree(jp);
        String data = node.textValue();
        return gson.parse(data);
    }
}

keycloak /auth page not found

with newer versions of keycloak, the `/auth` URL is no longer avaialble.

so instead of having OIDC_CLIENT_SECRETS as

{
  "web": {
    "issuer": "http://localhost:8080/auth/realms/flask-app",
    "auth_uri": "http://localhost:8080/auth/realms/flask-app/protocol/openid-connect/auth",
    "client_id": "flask-app",
    "client_secret": "XlBGtHzPOefRKjiEB9yTcQS0WBHllAcx",
    "redirect_uris": [
      "http://localhost:5000/*"
    ],
    "userinfo_uri": "http://localhost:8080/auth/realms/flask-app/protocol/openid-connect/userinfo",
    "token_uri": "http://localhost:8080/auth/realms/flask-app/protocol/openid-connect/token",
    "token_introspection_uri": "http://localhost:8080/auth/realms/flask-app/protocol/openid-connect/token/introspect"
  }
}

it should be

{
"web": {
"issuer": "http://localhost:8080/realms/flask-app",
"auth_uri": "http://localhost:8080/realms/flask-app/protocol/openid-connect/auth",
"client_id": "flask",
"client_secret": "pww8bSrx1sYC1oLsDkWwj7P2SS6BbpiK",
"redirect_uris": [
"http://localhost:5000/*"
],
"userinfo_uri": "http://localhost:8080/realms/flask-app/protocol/openid-connect/userinfo",
"token_uri": "http://localhost:8080/realms/flask-app/protocol/openid-connect/token",
"token_introspection_uri": "http://localhost:8080/realms/flask-app/protocol/openid-connect/token/introspect",
"bearer_only": "true"
}
}

JOL: java memory tooling

another great tool being used when i am troubleshoting the large memory foot print, together with

https://lwpro2.wordpress.com/2022/06/24/protobuf-memory-issue/

is JOL, which similar to instrumentation, however, providing that capability as a library, provide a great insight on the class and objects of being potential culprit.

to look at the class, or the shallow size of the memory foot print

ClassLayout.parseClass(A.class).toPrintable()

this would give the shallow size, or the object its own memory consumption, (where object variable calculated with the size of the memory reference itself alone)

to get the deep size, (which is the object size itself plus the object size being referred by this object)

GraphLayout.parseInstance(a).toFootPrint()
GraphLayout.parseInstance(a).toPrintable()

this gives

so here the payload size is 352 byte (the shallow size) + the object it has referred, total to 1640 byte.

Notice the unknownFiledSet now become 16 byte only, and TreeMap with only 48 byte, with the enhancement from

https://lwpro2.wordpress.com/2022/06/24/protobuf-memory-issue/

https://dzone.com/articles/java-object-size-estimation-measuring-verifying

protobuf memory issue

in addition to switch to zgc, https://lwpro2.wordpress.com/2022/06/23/jvm-zgc/.

the application still face OOM issue after running for sometime. After analyzing the heapdump, turns out there is an issue with the protobuf message.

The application is subscribing two kafka topics, one publishing message in JSON format, the second in protobuf.

Both messages could contain up to 700 fields for each single message. This is not an issue with the consumer for the json topic. At the parse of the kafka binary message, it could ignore all unknown fields. This is a feature of jackson, but almost always turned on by default. Because the real feature is called FAIL_ON_UNKNOWN_PROPERTIES, it's actually intended for not breaking the parse with unknown.

with this turned on (it has to in order for the parser to work), it resulted in a good state where the unknown fields are not eating up the JVM heap.

Out of the ~700 fields, it’s only ~30 fields needed in the application at the moment. Not all kafka message contain all 700 fields, sometimes it could contain 100 fields, sometimes contain much more. But even in the case it send over 100 fields over wire, jackson could correctly discard the 70 fields without polluting the memory.

This is however not the case for protobuf, at least not the default set up.

So with the protobuf topic also onboarded to the application, even only 30 fields out of each message is really needed, it could introduce x2(at a minimum) times garbage onto heap. And the thing with protobuf is, because the 30 fields are really needed, the protobuf class would be kept on the heap, eventually in the old gen, and survive many gc cycles (even some mixed and full gc).

ultimately the solution is to leverage on a trick, where the Message could provide a parser, which then could be wrapped by `DiscardUnknownFieldParser`, which then could be used for parse the binary.

so instead of

<T extends Message>.parseFrom(<binary byte[]>)

it could be

Parser<T extends Message> parser = DiscardUnknownFieldParser.wrap(<T extends Message>.parser());
parser.parseFrom(<binary byte[]>);

With both kafka messages, and the wrapped parser:

in addition, the ZGC comes with uncommit turned on by default, which would return the unused memory to host OS. so even though the application could have a very large Xmx, however, if it’s not used after gc, it would return that back to OS.

so with a docker allocated 450GB memory, the java application with 420GB Xmx, 180GB for Xms, its only consuming less than 30GB out of the OS.

ZGC wont uncommit memory more than Xms.

jmx to java process running out of docker

the trick to make the jmx work with the java process running out docker is to append the

`java.rmi.server.hostname` with the HOST (the box) docker is running on top of.

-Dcom.sun.management.jmxremote
-Dcom.sun.management.jmxremote.authenticate=false
-Dcom.sun.management.jmxremote.ssl=false
-Dcom.sun.management.jmxremote.port=<PORT>
-Dcom.sun.management.jmxremote.rmi.port=<PORT>
-Djava.rmi.server.hostname=$HOSTNAME

the $HOSTNAME can be passed in runtime

this would make visualVM work

otherwise, it could complain `Unsupported jdk`

class cast exception with spring ignite

when using ignite to return the object from cache (ignite), there was an exception thrown

turns out this an issue with the spring devtools, where while there is a base class loader, the devtools has created a split/new class loader for any exploded/inIDE classes.

somehow, when ignite (or any other deserializer according to https://github.com/spring-projects/spring-boot/commit/9a666d33662151c4cd30b1681cfcd36dccdc7eec#diff-d595239b5c62998cef5cd8a0103e65c2c4d6673a27ef115b1120220b5ec43ee2R1089) deserialize the bytes and load the class it was using the base class loader, hence cannot cast to the one loaded from devtools.

spring scheduler

i have a new app, which leverage on spring boot scheduler to start some cron jobs/process.

After running it for sometime, seems like the scheduler is no longer running on time. (Supposed to be run every 10 minutes on weekday).

Looked into this further, looks like this is the issue with the bulk head design pattern, where it leverage its own task pool for execution. and by default, spring made the task pool size 1.

so when one scheduler started a synchronous process, which took longer, it would cascade the effect for remaining schedulers, and all remaining become stuck.

the solution is to put back the custom scheduler i have implemented before for another application:

Supervised machine learning

libsvm is the first supervised machine learning library i have used extensively, more than 10 years back.

It was pretty awesome that time back, seeing a 78% text classification accuracy of against more than 100,000 hotel reviews, i have crawled from ctrip.com.

While, at version 3, they are able to achieve 96.875% for text classification results now, as:

Click to access guide.pdf

https://www.csie.ntu.edu.tw/~cjlin/libsvm/

JVM 7

http://docs.oracle.com/javase/7/docs/webnotes/tsg/TSG-VM/html/memleaks.html

3.1.1 Detail Message: Java heap space

The detail message Java heap space indicates that an object could not be allocated in the Java heap. This error does not necessarily imply a memory leak. The problem can be as simple as a configuration issue, where the specified heap size (or the default size, if not specified) is insufficient for the application.

In other cases, and in particular for a long-lived application, the message might be an indication that the application is unintentionally holding references to objects, and this prevents the objects from being garbage collected. This is the Java language equivalent of a memory leak. Note that APIs that are called by an application could also be unintentionally holding object references.

3.1.2 Detail Message: PermGen space

The detail message PermGen space indicates that the permanent generation is full. The permanent generation is the area of the heap where class and method objects are stored. If an application loads a very large number of classes, then the size of the permanent generation might need to be increased using the -XX:MaxPermSize option.

Interned java.lang.String objects are no longer stored in the permanent generation. The java.lang.String class maintains a pool of strings. When the intern method is invoked, the method checks the pool to see if an equal string is already in the pool. If there is, then the intern method returns it; otherwise it adds the string to the pool. In more precise terms, the java.lang.String.intern method is used to obtain the canonical representation of the string; the result is a reference to the same class instance that would be returned if that string appeared as a literal.

When this kind of error occurs, the text ClassLoader.defineClass might appear near the top of the stack trace that is printed.