Recently, I have implemented solution leveraging on protobuf.
The message are something like
message PositionGroup {
repeated Position payloads = 1;
}
message Position {
.....
}
There are several messages each with around 1 million repeated fields (the payloads). While the front tier (a graphql) return the combined result (expecting several million of records back), however, always return 1 million fields.
It took me quite a while to troubleshoot till surprisingly found out the protobuf implementation is totally different from what the API (mergeFrom) sounds like, make me almost feel like it’s a prank.
Per spec, “mergeFrom(Message other): (builder only) merges the contents of other into this message, overwriting singular scalar fields, merging composite fields, and concatenating repeated fields” so it does seem that the repeated payloads fields should be concatenated instead of replaced.
lwpro2.wordpress.com 