java redis OOM during serialization

I have a list (15 million) of POJOs, when put these onto redis, it will always throw out a OOM.

In the beginning, i thought it might be the heap size, however, after increasing that to even 100GB (the 15 million rows consuming ~30GB or less memory footprints), it still throw the same exception.

Looking further into that, seems like this is an issue due to the redis java client implementation which is taking in a varargs paramter.

the ArrayList is treated as an Object[] of size 1. (this is jdk’s implementation for varargs). However, when redis client trying to push that value onto the redis server, it’s reallying serializing one object (the arraylist of 15 million POJOs)

https://github.com/spring-projects/spring-data-redis/blob/main/src/main/java/org/springframework/data/redis/core/AbstractOperations.java#L136

somehow, for both jackson and default jdk serializer, it’s maintaining the size of bytes already written. with enough elements written, the number of bytes written would exceed `Integer.MAX_VALUE`

https://github.com/FasterXML/jackson-core/blob/2.14/src/main/java/com/fasterxml/jackson/core/util/ByteArrayBuilder.java#L279

Looks like redis client should parse the input differently if the input is a collection, instead of treating it as Object[]

gitlab issues

lately, I have spent a lot of time setting up GitLab pipelines. A lot of those time spent, now looking back, are actually wasted in the end, because turned out they’re bugs with gitlab itself.

  1. no matching artifacts
somehow, even with gitlab 14.8, this issue still exists. to make things even bizarre, it always works from the main (develop), however, on and off would throw alert "no matching files" from any merge request pipeline.
this is the setup:

build:
 stage: build
 script: 
  - mvn package
  - find . -type f -path '*/target/*.jar' # this shows the list of jars
 artifacts:
   paths:
      - ./*/target/*.jar'  # this only works in main branch, breaks on and off from other pipelines

https://gitlab.com/gitlab-org/gitlab-foss/-/issues/15530#note_889058639

https://gitlab.com/gitlab-org/gitlab/-/issues/36706#note_888550944

2. the way how gitlab handles timestamp in artifacts

I have tripped over and spent a lot of time on this as well.

But seems like the completion solution, is a combination of 
1) use new timestamp for *.class
2) disable the useIncrementalCompilation

at the moment, `useIncrementalCompilation` seems like only working from the pom.xml configuration, not through the command line.

Also it would be good if there is a parameter from gitlab to configure whether to reload the artifacts, or keep the existing timestamps.

```
build:
  stage: build
  image: maven:3.6-jdk-11
  script:
    - 'mvn test-compile'
    - *prepare-build-artifact
  except:
    - tags
  artifacts:
    paths:
      - target.zip
    reload: true
```

https://gitlab.com/gitlab-org/gitlab/-/issues/234078#note_885704266

git-commit-id-plugin auth fail

for some old version of git-commit-id-plugin, for example, `

4.9.10

there is a Auth Fail error, when the plugin is in execution.

Turns out there was a bug (should be fixed in latest version), which uses JGitDataProvider, and requesting for

`

LOCAL_BRANCH_AHEAD and LOCAL_BRANCH_BEHIND

mark the plugin as office sorted out the issue

            <plugin>
                <groupId>pl.project13.maven</groupId>
                <artifactId>git-commit-id-plugin</artifactId>
                <version>4.9.10</version>
                <configuration>
                    <verbose>false</verbose>
                    <useNativeGit>true</useNativeGit>
                    <offline>true</offline>
                </configuration>

spring configuration properties

Turns out when spring boot construct the configuration bean, it’s using the setter method to map the values.

So for IgniteConfiguration, for example, even though the variable for peer ClassLoading is

private boolean p2pEnabled = DFLT_P2P_ENABLED;

however, the values below won’t work

p2pEnabled:
  true

instead, it should be

peerClassLoadingEnabled: true

as the setter to set the p2pEnabled is actually called 

    public IgniteConfiguration setPeerClassLoadingEnabled(boolean p2pEnabled) {
        this.p2pEnabled = p2pEnabled;

        return this;
    }

JVM memory tuning with Docker

I have several applications that were running with 450GB memory before. It did seem like it’s a valid memory consumption, after manually calculating the memory footprint, specifically the sum up of all primitive types and object sizes.

however, the same applications are now able to run with ~50GB from a docker container.

looks like the misalignment is coming from the underestimate of the JVM’s intelligence at the moment.

A little background here, it has been pretty common now in the market, in general, to set the -Xmx and -Xms parameters manually at the application startup. but seems like JVM itself is able to do a better job than developers’ estimate for the heap size.

I have been tuning some of the applications continuously for quite some, up and down some of the applications of the `-Xmx` from 300GB to 400GB to 5-600GB, then finalized at 450GB.

while at the moment, together with string deduplication, which is an important factor as there are a lot of repeated strings like currency, security, and PM names; and the docker set up where I have set up a memory limit, and dropping out the -Xmx parameter completely, it seems like these actions together made the JVM shine its capabilities, that it probably have kicked the GC more intelligently/aggressively, and the `-Xmx` has been calculated more accurately by JVM.