MODERN ARCHITECTURAL PRINCIPLES FOR DESIGNING HIGH-PERFORMANCE DISTRIBUTED SYSTEMS

Ivan Rovetskii

Abstract


Background. The shift to online businesses has driven unprecedented growth in traffic density and data volumes. Engineering scalable high-load systems involves challenges like systemic latency, cascading failures from tight component coupling, I/O boundaries and CAP theorem trade-offs. Flawed architectural patterns cause performance degradation and financial losses. Therefore, the goal of this study is to analyze and systematize architectural solutions to minimize mentioned infrastructure challenges and ensure linear scalability of high-load applications.

Materials and Methods. This paper presents a theoretical study on the architectural design of high-load distributed systems based on Apache Kafka. The research synthesizes a comparative analysis of academic literature with the author's practical development experience.

Results and Discussion. A theoretical analysis reveals that synchronous inter-service communication based on HTTP/gRPC protocols leads to reduced efficiency in high-load environments due to cascading delays, tight coupling and thread pool exhaustion risks. The study substantiates that transitioning to an asynchronous Publish/Subscribe with a Pull-based model via Apache Kafka provides significant architectural advantages for extreme horizontal scalability. It provides a stable backpressure mechanism, prevents computational node overloading, minimizes CPU overhead through batch aggregation and simplifies broker architecture by shifting data offset persistence to the consumer. The paper systemically analyzes specific Apache Kafka engineering bottlenecks and proposes effective mitigation methods.

Conclusion. The paper analyzes architectural solutions for high-load systems based on the Apache Kafka platform, aimed at increasing the efficiency of stream data processing. The suggested approaches minimize the systemic risks of asynchronous communication, ensuring protection against event chronology violation in distributed transactions, data skew and anomalous message duplication.

Keywords: high-load distributed systems, CAP theorem, BASE model, Apache Kafka, backpressure, distributed transactions


Full Text:

PDF

References


[1] Newman, S. (2021). Building microservices: Designing fine-grained systems (2nd ed.). O’Reilly Media.

[2] Kleppmann, M. (2017). Designing Data-Intensive Applications: The Big Ideas Behind Reliable, Scalable, and Maintainable Systems. O'Reilly Media.

[3] Kreps, J., Narkhede, N., & Rao, J. (2011). Kafka: A distributed messaging system for log processing. In Proceedings of the 6th International Workshop on Networking Meets Databases (NetDB 2011) (pp. 1–7). https://people.csail.mit.edu/matei/courses/2015/6.S897/readings/kafka.pdf

[4] Özsu, M. T., & Valduriez, P. (2020). Principles of Distributed Database Systems (4th ed. pp. 183–246). Springer. https://doi.org/10.1007/978/3-030-26253-2

[5] Richardson, C. (2018). Microservices Patterns: With examples in Java. Manning Publications.

[6] Narkhede, N., Shapira, G., & Palino, T. (2017). Kafka: The Definitive Guide: Real-Time Data and Stream Processing at Scale. O'Reilly Media.

[7] Hohpe, G., & Woolf, B. (2003). Enterprise Integration Patterns: Designing, Building, and Deploying Messaging Solutions. Addison-Wesley.

[8] Evans, E. (2003). Domain-Driven Design: Tackling Complexity in the Heart of Software. Addison-Wesley.

[9] Garcia-Molina, H., & Salem, K. (1987). Sagas. In Proceedings of the 1987 ACM SIGMOD International Conference on Management of Data (pp. 249–259). ACM. https://doi.org/10.1145/38713.38742

[10] Brewer, E. A. (2000). Towards robust distributed systems (abstract). In Proceedings of the Nineteenth Annual ACM Symposium on Principles of Distributed Computing (p. 7). ACM. https://doi.org/10.1145/343477.343502

[11] Gilbert, S., & Lynch, N. (2002). Brewer’s conjecture and the feasibility of consistent, available, partition-tolerant web services. ACM SIGACT News, 33(2), 51–59. https://doi.org/10.1145/564585.564601

[12] Pritchett, D. (2008). BASE: An acid alternative. ACM Queue, 6(3), 48–55. https://doi.org/10.1145/1394127.1394128

[13] Henning, S., & Hasselbring, W. (2024). Benchmarking scalability of stream processing frameworks deployed as microservices in the cloud. The Journal of Systems and Software, 208, 111879. https://doi.org/10.1016/j.jss.2023.111879

[14] Wang, G., Chen, L., Dikshit, A., Gustafson, J., Chen, B., Sax, M. J., Roesler, J., Blee-Goldman, S., Cadonna, B., Mehta, A., Madan, V., & Rao, J. (2021). Consistency and completeness: Rethinking distributed stream processing in Apache Kafka. In Proceedings of the 2021 International Conference on Management of Data (pp. 2602–2613). ACM. https://doi.org/10.1145/3448016.3457556

[15] Menda, J. R. (2018). Real-time financial settlement using Kafka Streams and Cassandra: A distributed architecture for low latency, exactly-once processing. Journal of Scientific and Engineering Research, 5(10), 362-372. https://doi.org/10.5281/zenodo.18084995

[16] Bonér, J. (2016). Reactive microservices architecture: Design principles for distributed systems. O’Reilly Media. https://www.oreilly.com/library/view/reactive-microservices-architecture/9781491975664/

[17] Vernon, V. (2023). Strategic Monolith-to-Microservices Migration Using Domain-Driven Design and Event Streaming. Addison-Wesley.

[18] Vogel, A., Henning, S., Pérez-Wohlfeil, E., Ertl, O., & Rabiser, R. (2024). A comprehensive benchmarking analysis of fault recovery in stream processing frameworks. In Proceedings of the 18th ACM International Conference on Distributed and Event-Based Systems (pp. 171–182). https://doi.org/10.1145/3629104.3666040

[19] Stopford, B. (2024). Designing Event-Driven Systems: Concepts and Patterns for Streaming Services with Apache Kafka. O’Reilly Media.

[20] Videla, A., & Williams, J. (2012). RabbitMQ in action: Distributed messaging for everyone. Manning Publications.




DOI: http://dx.doi.org/10.30970/eli.35.5

Refbacks

  • There are currently no refbacks.