Apache RocketMQ Files

What's Changed

This release introduces a new implementation of Pop consumption based on RocksDB (in the alpha phase, see RIP-73 for details). It improves and provides a migration plan for the capability of using RocksDB to store consumption queue indexes, adds support for the ability to cancel scheduled messages, enhances graceful online and offline capabilities, and includes numerous regular detail fixes related to HA, ACL, Metrics and Pop consumption.

• [ISSUE [#8784]] PrintMessageSubCommand support lmq by @RongtongJin in https://github.com/apache/rocketmq/pull/8785
• [ISSUE [#8790]] Add Utils for put header to Metadata to avoid duplicate data by @dingshuangxi888 in https://github.com/apache/rocketmq/pull/8792
• [ISSUE [#8802]] Update controller design.md by @luozongle01 in https://github.com/apache/rocketmq/pull/8803
• [ISSUE [#8798]] Fix typo by @luozongle01 in https://github.com/apache/rocketmq/pull/8799
• [ISSUE [#8804]] Clean offset when remove group offset by @leizhiyuan in https://github.com/apache/rocketmq/pull/8805
• [ISSUE [#8806]] Fix autoBatch bug when connecting multiple RocketMQ clusters. by @luozongle01 in https://github.com/apache/rocketmq/pull/8807
• [ISSUE [#8810]] Fix independent execution of e2e and benchmark deployments by @chi3316 in https://github.com/apache/rocketmq/pull/8812
• [ISSUE [#8816]] Fix log typo. by @luozongle01 in https://github.com/apache/rocketmq/pull/8817
• [ISSUE [#8764]] Implement consume lag estimation in cq rocksdb store by @LetLetMe in https://github.com/apache/rocketmq/pull/8800
• [ISSUE [#8824]] Fix IllegalStateException caused by logical errors by @yx9o in https://github.com/apache/rocketmq/pull/8825
• [ISSUE [#8780]] Implement asynchronous storage of ack/ck messages in pop consume to enhance performance by @RongtongJin in https://github.com/apache/rocketmq/pull/8727
• [ISSUE [#8835]] When ck is in the buffer, incomplete ack will lead to message duplication. by @GenerousMan in https://github.com/apache/rocketmq/pull/8836
• [ISSUE [#8820]] Fix variables match annotation (@RocketMQResource) by @mawen12 in https://github.com/apache/rocketmq/pull/8821
• [ISSUE [#8830]] Fix atomic flush incorrect use and clean up code by @lizhanhui in https://github.com/apache/rocketmq/pull/8830
• [ISSUE [#8829]] Fix make ConsumeQueueStore bottom most compression type configurable by @lizhanhui in https://github.com/apache/rocketmq/pull/8841
• [ISSUE [#8839]] Remove unnecessary synchronized to improve concurrency by @lizhanhui in https://github.com/apache/rocketmq/pull/8840
• [ISSUE [#8833]] Update netty version to 4.1.114 to fix CVE-2023-34462 by @ShannonDing in https://github.com/apache/rocketmq/pull/8832
• [ISSUE [#8846]] Fix registerProducer should not be affected by concurrent scanNotAct… by @lizhanhui in https://github.com/apache/rocketmq/pull/8847
• [ISSUE [#8829]] Support LMQ dispatch in case if Consume Queue Store is RocksDB-based by @lizhanhui in https://github.com/apache/rocketmq/pull/8842
• [ISSUE [#8850]] Use correct log method by @mawen12 in https://github.com/apache/rocketmq/pull/8851
• [ISSUE [#8772]] Remove lock mq step in broadcasting mode rebalancing by @redlsz in https://github.com/apache/rocketmq/pull/8774
• [ISSUE [#8442]] [RIP-70] Extract adaptive lock mechanism by @3424672656 in https://github.com/apache/rocketmq/pull/8663
• [ISSUE [#8829]] Feat: provide ConfigManagerV2 to make best uses of RocksDB by @lizhanhui in https://github.com/apache/rocketmq/pull/8856
• [ISSUE [#8822]] Double write cq, reduce unnecessary switches by @LetLetMe in https://github.com/apache/rocketmq/pull/8823
• [ISSUE [#8875]] Fix HAConnection leak by @crazywen in https://github.com/apache/rocketmq/pull/8876
• [ISSUE [#8725]] Clean DefaultMQPushConsumer after start fail by @yuz10 in https://github.com/apache/rocketmq/pull/8726
• [ISSUE [#8829]] Keep data version while reload and XXXConfigManagerV2 turns off sync by @lizhanhui in https://github.com/apache/rocketmq/pull/8891
• [ISSUE [#8885]] Resolve the issue of inaccurate CK number statistics by @RongtongJin in https://github.com/apache/rocketmq/pull/8886
• [ISSUE [#8808]] Resolve unsupported 'UseBiasedLocking' VM Option for JDK21 by @taomaree in https://github.com/apache/rocketmq/pull/8809
• [ISSUE [#8882]] Change the compare method for acl signature to improve the security. by @dingshuangxi888 in https://github.com/apache/rocketmq/pull/8883
• [ISSUE [#8889]] handle namespace outside the loop by @mawen12 in https://github.com/apache/rocketmq/pull/8890
• [ISSUE [#8906]] Handle string toUpperCase outside the loop by @mawen12 in https://github.com/apache/rocketmq/pull/8907
• [ISSUE [#8599]] Throw exception when receiving GO_AWAY twice to close channel. by @qianye1001 in https://github.com/apache/rocketmq/pull/8862
• [ISSUE [#8917]] Topic route return none permission message queues for gRPC client by @dingshuangxi888 in https://github.com/apache/rocketmq/pull/8919
• [ISSUE [#8921]] Add isWakeCommitWhenPutMessage for AIO by @zk-drizzle in https://github.com/apache/rocketmq/pull/8922
• [ISSUE [#8925]] Fix the bug of update user for ACL 2.0 by @dingshuangxi888 in https://github.com/apache/rocketmq/pull/8926
• [ISSUE [#8909]] Move nullcheck ahead by @jjastan in https://github.com/apache/rocketmq/pull/8910
• [ISSUE [#8942]] Add incGroupAckNums and incGroupCkNums to LmqBrokerStatsManager by @RongtongJin in https://github.com/apache/rocketmq/pull/8943
• [ISSUE [#8829]] Support for Persisting LMQ Consumer Offsets in Config V1 Using RocksDB by @RongtongJin in https://github.com/apache/rocketmq/pull/8939
• [ISSUE [#8945]] Remove unnecessary operations from the critical section by @3424672656 in https://github.com/apache/rocketmq/pull/8946
• [ISSUE [#8940]] Fix incorrect path for exportMetadataInRocksDBCommand by @RongtongJin in https://github.com/apache/rocketmq/pull/8941
• [ISSUE [#8935]] Fix behind metrics unit error in timer message store by @GenerousMan in https://github.com/apache/rocketmq/pull/8936
• [ISSUE [#8765]] Fix low performance of delay message when enable rocksdb consume queue by @yuz10 in https://github.com/apache/rocketmq/pull/8766
• [ISSUE [#8953]] Improve IO for asynchronous delivery processes by @3424672656 in https://github.com/apache/rocketmq/pull/8954
• [ISSUE [#8829]] Fix avoid memory overhead when there is large number of LMQ CQ by @lizhanhui in https://github.com/apache/rocketmq/pull/8956
• [ISSUE [#8947]] Notify pop request before calculate consumer lag by @lizhimins in https://github.com/apache/rocketmq/pull/8949
• [ISSUE [#8933]] feat: DefaultPullConsumer add balance switch. by @humkum in https://github.com/apache/rocketmq/pull/8934
• [ISSUE [#8929]] Proxy adds message body empty check when send in grpc protocol by @qianye1001 in https://github.com/apache/rocketmq/pull/8930
• [ISSUE [#8877]] Refactor lock in ReceiptHandleGroup to make the lock can be properly released when future can not be completed. by @qianye1001 in https://github.com/apache/rocketmq/pull/8916
• [ISSUE [#8955]] Fix message buffer not release and dispatch thread exit in tiered storage by @lizhimins in https://github.com/apache/rocketmq/pull/8965
• [ISSUE [#8972]] Adding the EnableLmqStats option allows monitoring of LMQ statistics at runtime by @RongtongJin in https://github.com/apache/rocketmq/pull/8973
• [ISSUE [#8961]] Automatic recognition of address scheme in Topic Route by host by @dingshuangxi888 in https://github.com/apache/rocketmq/pull/8962
• [ISSUE [#8460]] Set default broker name when revive found ack without broker name field by @lizhimins in https://github.com/apache/rocketmq/pull/8981
• [ISSUE [#8982]] Dynamically install latest Go version for e2e pipeline by @chi3316 in https://github.com/apache/rocketmq/pull/8985
• [ISSUE [#8976]] Modify file segment construct method by @lizhimins in https://github.com/apache/rocketmq/pull/8977
• [ISSUE [#8963]] Fix code36 request sent to ns by @yx9o in https://github.com/apache/rocketmq/pull/8964
• [ISSUE [#8968]] Introduce the clearRetryTopicWhenDeleteTopic option to enable precise external deletion of topics by @RongtongJin in https://github.com/apache/rocketmq/pull/8969
• [ISSUE [#8991]] PrepareHeartbeatData should not be set by default subscriptionDataSet data by @weihubeats in https://github.com/apache/rocketmq/pull/8992
• [ISSUE [#7199]] GrpcClientChannel header add null judgement by @weihubeats in https://github.com/apache/rocketmq/pull/7238
• [ISSUE [#9009]] Optimize message accumulation display and Metrics in DefaultPullConsumer by @qianye1001 in https://github.com/apache/rocketmq/pull/9010
• [ISSUE [#8984]] Fix the broker switch enableMixedMessageType doesn't work by @redlsz in https://github.com/apache/rocketmq/pull/8986
• [ISSUE [#8950]] Remove Redundant nullcheck of configPath by @jjastan in https://github.com/apache/rocketmq/pull/8951
• [ISSUE [#8966]] Feat: add remote address information to acl perm error by @humkum in https://github.com/apache/rocketmq/pull/8967
• [ISSUE [#9007]] Fix client connection local ip is null in RemotingClient by @weihubeats in https://github.com/apache/rocketmq/pull/9008
• [ISSUE [#9014]] Fix clusterAclConfigVersion command execution failed by @yx9o in https://github.com/apache/rocketmq/pull/9017
• [ISSUE [#7480]] Fix the offset in the timerCheckPoint will not be corrected when the commitlog and consumeQueue are truncated by @RongtongJin in https://github.com/apache/rocketmq/pull/7488
• [ISSUE [#9015]] Sync SysFlag and message body inflation status; allow omit of message body by @lizhanhui in https://github.com/apache/rocketmq/pull/9016
• [ISSUE [#8979]] Add configurable switch for timer message retry logic by @chi3316 in https://github.com/apache/rocketmq/pull/8980
• [ISSUE [#9034]] Refactor(LmqBrokerStatsManager): Refactor Redundant Code in LmqBrokerStatsManager by @asapple in https://github.com/apache/rocketmq/pull/9034
• [ISSUE [#8974]] Support recalling of delay message by @imzs in https://github.com/apache/rocketmq/pull/8975
• [ISSUE [#9042]] Update createTimerMessageStore call with new parameter by @chi3316 in https://github.com/apache/rocketmq/pull/9041
• [ISSUE [#9021]] Correct the error message of acl command by @yx9o in https://github.com/apache/rocketmq/pull/9022
• [ISSUE [#8970]] Remove redundant heartbeats by @weihubeats in https://github.com/apache/rocketmq/pull/8971
• [ISSUE [#8988]] Support dispatchBehindMilliseconds by @guyinyou in https://github.com/apache/rocketmq/pull/8989
• [ISSUE [#9028]] Adjust some error code for SYSTEM_ERROR by @absolute8511 in https://github.com/apache/rocketmq/pull/9027
• [ISSUE [#9054]] Optimize log print when client consume message in pop mode by @mxsm in https://github.com/apache/rocketmq/pull/9055
• [ISSUE [#9002]] When bytebuffer is not enough, do not throw exception by @leizhiyuan in https://github.com/apache/rocketmq/pull/9003
• [ISSUE [#9072]] Fix the permission check for retry topic to get topic route. by @dingshuangxi888 in https://github.com/apache/rocketmq/pull/9073
• [ISSUE [#9025]] [RIP-73] Pop Consumption Improvement Based on RocksDB by @lizhimins in https://github.com/apache/rocketmq/pull/9048
• [ISSUE [#8957]] Remove excess traffic and fix cache inconsistencies by @3424672656 in https://github.com/apache/rocketmq/pull/8958
• [ISSUE [#9069]] Fix the IndexFile ConcurrentModificationException in tiered storage by @wangshaojie4039 in https://github.com/apache/rocketmq/pull/9071
• [ISSUE [#9075]] Avoid message type validate in message sync scenario. by @dingshuangxi888 in https://github.com/apache/rocketmq/pull/9076
• [ISSUE [#9080]] Fix tranfer logic when get large messages from cache in tiered storage by @yuz10 in https://github.com/apache/rocketmq/pull/9079
• [ISSUE [#8974]] Add feature switch of recalling, disable by default. by @imzs in https://github.com/apache/rocketmq/pull/9067
• [ISSUE [#9080]] Not hold final message store config in fetcher by @lizhimins in https://github.com/apache/rocketmq/pull/9086
• [ISSUE [#9025]] [RIP-73] Fix reset offset not effective in Pop Consumption by @lizhimins in https://github.com/apache/rocketmq/pull/9087
• [ISSUE [#9025]] [RIP-73] Modify Pop Consumption rocksdb init config by @lizhimins in https://github.com/apache/rocketmq/pull/9100
• [ISSUE [#9106]] Fix revive backoff retry not effective in Pop Consumption based on rocksdb by @redlsz in https://github.com/apache/rocketmq/pull/9107
• [ISSUE [#8998]] No retry is required when the remaining time reaches zero by @3424672656 in https://github.com/apache/rocketmq/pull/8999
• [ISSUE [#9105]] Fix the issue of duplicate consumption in LMQ by @RongtongJin in https://github.com/apache/rocketmq/pull/9101
• [ISSUE [#9108]] Refactor ColdDataCgCtrService#getColdDataFlowCtrInfo by @yx9o in https://github.com/apache/rocketmq/pull/9109
• [ISSUE [#9064]] Optimize transaction message callback check logic by @LilMosey in https://github.com/apache/rocketmq/pull/9062
• [ISSUE [#9112]] Speedup revive scan in Pop Consumption and support server side reset offset by @lizhimins in https://github.com/apache/rocketmq/pull/9113
• [ISSUE [#9121]] Fix CRC32 Check Failing When Value is 0 by @guyinyou in https://github.com/apache/rocketmq/pull/9122
• [ISSEU [#6426]] Fix slave broker SubscriptionGroupConfig and MessageRequestMode updating atomically by @AuroraTwinkle in https://github.com/apache/rocketmq/pull/8983
• [ISSUE [#9111]] Support export broker RocksDB Config to json file by @qianye1001 in https://github.com/apache/rocketmq/pull/9114
• [ISSUE [#8895]] Fix NPE when broker shutdown and optimize the log by @qianye1001 in https://github.com/apache/rocketmq/pull/9094
• [ISSUE [#9128]] Fix NPE when gRPC client ack message immediately after changing proxy by @qianye1001 in https://github.com/apache/rocketmq/pull/9129
• [ISSUE [#4570]] Fix: Docker usage may occur error in volume mapping params, simple fix by @OneCodeMonkey in https://github.com/apache/rocketmq/pull/9096
• [ISSUE [#9119]] Invoke async should handle raw exception instead of CompletionException by @gaoyf in https://github.com/apache/rocketmq/pull/9120
• [ISSUE [#9160]] Ensure the requestCode increments sequentially by @KiteSoar in https://github.com/apache/rocketmq/pull/9159
• [ISSUE [#9152]] Broker getConsumeStats supports inputting multiple topics by @qianye1001 in https://github.com/apache/rocketmq/pull/9153
• [ISSUE [#9149]] Assign offset in offsetTable even if the subscription key not exist. by @dingshuangxi888 in https://github.com/apache/rocketmq/pull/9150
• [ISSUE [#9156]] Use fastjson2 in AclUtils#getAclRPCHook by @yx9o in https://github.com/apache/rocketmq/pull/9157
• [ISSUE [#9174]] Add a collection of predefined Groups and common che… by @ltamber in https://github.com/apache/rocketmq/pull/9175
• [ISSUE [#9172]] Clean pull offset and reset offset when delete subscription group by @lizhimins in https://github.com/apache/rocketmq/pull/9173
• [ISSUE [#9181]] Update fastjson version by @ltamber in https://github.com/apache/rocketmq/pull/9180
• [ISSUE [#9182]] Fix NameServer will be not ready forever when set needWaitForService to true by @gaoyf in https://github.com/apache/rocketmq/pull/9183
• [ISSUE [#9187]] The request should be rejected if the queueOffset equals maxOffset when changing the invisible time by @RongtongJin in https://github.com/apache/rocketmq/pull/9186
• [ISSUE [#9201]] Cleanup dead code patterns and improve readability and maintainability. by @Kris20030907 in https://github.com/apache/rocketmq/pull/9202
• [ISSUE [#9191]] Provide the ability to replace the remoting layer implementation for Proxy and Broker by @f1amingo in https://github.com/apache/rocketmq/pull/9192
• [ISSUE [#9203]] Replace numbers with static variables defined in RequestCode by @xinzhuxiansheng in https://github.com/apache/rocketmq/pull/9204
• [ISSUE [#9206]] Fix slave sync topic sub in rocksdb ha by @fujian-zfj in https://github.com/apache/rocketmq/pull/9207
• [ISSUE [#8340]] RuntimeInfo and ClusterListSubCommand show ackThreadPoolQueueSize and ackThreadPoolQueueHeadWaitTimeMills by @RongtongJin in https://github.com/apache/rocketmq/pull/8339
• [ISSUE [#8589]] Optimize RocksDB CQ shutdown when using DoubleWriteCQ by @qianye1001 in https://github.com/apache/rocketmq/pull/9212
• [ISSUE [#9213]] Fix get the earliest time error when data is clean up in tiered storage by @lizhimins in https://github.com/apache/rocketmq/pull/9214
• [ISSUE [#9196]] Broker return pop stats when receive notification by @qianye1001 in https://github.com/apache/rocketmq/pull/9197
• [ISSUE [#9217]] Fix broker's inflight and available message counts incorrect when the pop consumer service is enabled by @lizhimins in https://github.com/apache/rocketmq/pull/9218
• [ISSUE [#9221]] Extract some common code in BrokerPathConfigHelper by @yx9o in https://github.com/apache/rocketmq/pull/9222
• [ISSUE [#8127]] Optimize the metric calculation logic of the time wheel by @3424672656 in https://github.com/apache/rocketmq/pull/8128
• [ISSUE [#9184]] Optimize QueueLockManager#tryLock method in Pop by @mxsm in https://github.com/apache/rocketmq/pull/9185

New Contributors

• @mawen12 made their first contribution in https://github.com/apache/rocketmq/pull/8821
• @taomaree made their first contribution in https://github.com/apache/rocketmq/pull/8809
• @jjastan made their first contribution in https://github.com/apache/rocketmq/pull/8910
• @asapple made their first contribution in https://github.com/apache/rocketmq/pull/9034
• @draw233 made their first contribution in https://github.com/apache/rocketmq/pull/9037
• @LilMosey made their first contribution in https://github.com/apache/rocketmq/pull/9062
• @AuroraTwinkle made their first contribution in https://github.com/apache/rocketmq/pull/8983
• @OneCodeMonkey made their first contribution in https://github.com/apache/rocketmq/pull/9096
• @Kris20030907 made their first contribution in https://github.com/apache/rocketmq/pull/9124
• @KiteSoar made their first contribution in https://github.com/apache/rocketmq/pull/9159
• @Lynxhide made their first contribution in https://github.com/apache/rocketmq/pull/9158
• @xinzhuxiansheng made their first contribution in https://github.com/apache/rocketmq/pull/9204

Full Changelog: https://github.com/apache/rocketmq/compare/rocketmq-all-5.3.1...rocketmq-all-5.3.2