Vol. 1 No. 1 (2021): African Journal of Artificial Intelligence and Sustainable Development
Articles

Enhanced Logging and Monitoring with Custom Metrics in Kubernetes

PDF
  • Babulal Shaik,
  • Jayaram Immaneni
Babulal Shaik
Cloud Solutions Architect at Amazon Web Services, USA
Jayaram Immaneni
SRE Lead at JP Morgan Chase, USA
Cover

Published 03-04-2021

Keywords

  • Kubernetes,
  • Elastic Kubernetes Service (EKS),
  • logging
Creative Commons License

This work is licensed under a Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International License.

How to Cite

[1]
Babulal Shaik and Jayaram Immaneni, “Enhanced Logging and Monitoring with Custom Metrics in Kubernetes ”, African J. of Artificial Int. and Sust. Dev., vol. 1, no. 1, pp. 307–330, Apr. 2021, Accessed: Dec. 29, 2024. [Online]. Available: https://africansciencegroup.com/index.php/AJAISD/article/view/220

Abstract

Kubernetes has revolutionized how we deploy and manage containerized applications, but with its dynamic and distributed nature, effective logging and monitoring are critical to maintaining system health and performance. Traditional monitoring approaches often need to address the unique challenges Kubernetes poses, such as temporary containers, autoscaling, and the complexity of microservices. Organizations are adopting enhanced logging and monitoring techniques to bridge this gap, enriched with custom metrics, to gain deeper insights into their clusters. Custom metrics allow teams to tailor monitoring solutions to their application needs beyond generic system-level metrics like CPU and memory usage. This enables proactive detection of anomalies, fine-grained performance tracking, and a better understanding of application behaviour in real-time. By integrating tools such as Prometheus, Grafana, Fluentd, and open-source exporters, Kubernetes users can create a seamless pipeline for metric collection, visualization, and alerting. Coupled with structured logging and centralized log aggregation, these enhancements simplify debugging and improve the observability of complex, multi-service environments. This approach enhances system reliability and empowers DevOps teams to implement data-driven optimizations, ensuring smoother operations and more resilient applications. For organizations leveraging Kubernetes in production, mastering these advanced logging and monitoring strategies is essential to maintaining high availability and achieving operational excellence in an ever-evolving cloud-native landscape.

PDF

Downloads

Download data is not yet available.

References

  1. Ritari, O. (2019). Monitoring a Kubernetes Application.
  2. Kubernetes, T. (2019). Kubernetes. Kubernetes. Retrieved May, 24, 2019.
  3. Chiba, T., Nakazawa, R., Horii, H., Suneja, S., & Seelam, S. (2019, June). Confadvisor: A performance-centric configuration tuning framework for containers on kubernetes. In 2019 IEEE International Conference on Cloud Engineering (IC2E) (pp. 168-178). IEEE.
  4. Sayfan, G. (2018). Mastering Kubernetes: Master the art of container management by using the power of Kubernetes. Packt Publishing Ltd.
  5. Burns, B., & Tracey, C. (2018). Managing Kubernetes: operating Kubernetes clusters in the real world. O'Reilly Media.
  6. Oliveira, F., Suneja, S., Nadgowda, S., Nagpurkar, P., & Isci, C. (2017). A cloud-native monitoring and analytics framework. IBM Research Division Thomas J. Watson Research Center, Tech. Rep. RC25669 (WAT1710-006), 119.
  7. Bastos, J., & Araújo, P. (2019). Hands-On Infrastructure Monitoring with Prometheus: Implement and scale queries, dashboards, and alerting across machines and containers. Packt Publishing Ltd.
  8. Luksa, M. (2017). Kubernetes in action. Simon and Schuster.
  9. Shemyakinskaya, A. S., & Nikiforov, I. V. (2020). Hard drives monitoring automation approach for Kubernetes container orchestration system. Труды института системного программирования РАН, 32(2), 99-106.
  10. Carcassi, G., Breen, J., Bryant, L., Gardner, R. W., McKee, S., & Weaver, C. (2020). SLATE: Monitoring distributed Kubernetes clusters. In Practice and Experience in Advanced Research Computing (pp. 19-25).
  11. Katari, A., & Rallabhandi, R. S. DELTA LAKE IN FINTECH: ENHANCING DATA LAKE RELIABILITY WITH ACID TRANSACTIONS.
  12. Gade, K. R. (2018). Real-Time Analytics: Challenges and Opportunities. Innovative Computer Sciences Journal, 4(1).
  13. Shiraishi, T., Noro, M., Kondo, R., Takano, Y., & Oguchi, N. (2020, September). Real-time monitoring system for container networks in the era of microservices. In 2020 21st Asia-Pacific Network Operations and Management Symposium (APNOMS) (pp. 161-166). IEEE.
  14. Kothapalli, K. R. V. (2019). Enhancing DevOps with Azure Cloud Continuous Integration and Deployment Solutions. Engineering International, 7(2), 179-192.
  15. Larghi, F. (2018). LLAMA. A system for log management and analysis on a complex distributed environment.
  16. Immaneni, J. (2020). Cloud Migration for Fintech: How Kubernetes Enables Multi-Cloud Success. Innovative Computer Sciences Journal, 6(1).
  17. Boda, V. V. R., & Immaneni, J. (2019). Streamlining FinTech Operations: The Power of SysOps and Smart Automation. Innovative Computer Sciences Journal, 5(1).
  18. Nookala, G., Gade, K. R., Dulam, N., & Thumburu, S. K. R. (2020). Data Virtualization as an Alternative to Traditional Data Warehousing: Use Cases and Challenges. Innovative Computer Sciences Journal, 6(1).
  19. Nookala, G., Gade, K. R., Dulam, N., & Thumburu, S. K. R. (2019). End-to-End Encryption in Enterprise Data Systems: Trends and Implementation Challenges. Innovative Computer Sciences Journal, 5(1).
  20. Komandla, V. Enhancing Security and Fraud Prevention in Fintech: Comprehensive Strategies for Secure Online Account Opening.
  21. Komandla, V. Transforming Financial Interactions: Best Practices for Mobile Banking App Design and Functionality to Boost User Engagement and Satisfaction.
  22. Thumburu, S. K. R. (2020). Enhancing Data Compliance in EDI Transactions. Innovative Computer Sciences Journal, 6(1).
  23. Thumburu, S. K. R. (2020). Leveraging APIs in EDI Migration Projects. MZ Computing Journal, 1(1).
  24. Gade, K. R. (2020). Data Mesh Architecture: A Scalable and Resilient Approach to Data Management. Innovative Computer Sciences Journal, 6(1).
  25. Gade, K. R. (2020). Data Analytics: Data Privacy, Data Ethics, Data Monetization. MZ Computing Journal, 1(1).
  26. Katari, A. (2019). Real-Time Data Replication in Fintech: Technologies and Best Practices. Innovative Computer Sciences Journal, 5(1).
  27. Katari, A. (2019). ETL for Real-Time Financial Analytics: Architectures and Challenges. Innovative Computer Sciences Journal, 5(1).
  28. Gade, K. R. (2017). Integrations: ETL vs. ELT: Comparative analysis and best practices. Innovative Computer Sciences Journal, 3(1).
  29. Muneer Ahmed Salamkar. Batch Vs. Stream Processing: In-Depth Comparison of Technologies, With Insights on Selecting the Right Approach for Specific Use Cases. Distributed Learning and Broad Applications in Scientific Research, vol. 6, Feb. 2020
  30. Muneer Ahmed Salamkar, and Karthik Allam. Data Integration Techniques: Exploring Tools and Methodologies for Harmonizing Data across Diverse Systems and Sources. Distributed Learning and Broad Applications in Scientific Research, vol. 6, June 2020
  31. Naresh Dulam, et al. “Data As a Product: How Data Mesh Is Decentralizing Data Architectures”. Distributed Learning and Broad Applications in Scientific Research, vol. 6, Apr. 2020
  32. Naresh Dulam, et al. “Data Mesh in Practice: How Organizations Are Decentralizing Data Ownership ”. Distributed Learning and Broad Applications in Scientific Research, vol. 6, July 2020
  33. Sarbaree Mishra. “Moving Data Warehousing and Analytics to the Cloud to Improve Scalability, Performance and Cost-Efficiency”. Distributed Learning and Broad Applications in Scientific Research, vol. 6, Feb. 2020
  34. Sarbaree Mishra, et al. “Training AI Models on Sensitive Data - the Federated Learning Approach”. Distributed Learning and Broad Applications in Scientific Research, vol. 6, Apr. 2020
  35. Babulal Shaik. Network Isolation Techniques in Multi-Tenant EKS Clusters. Distributed Learning and Broad Applications in Scientific Research, vol. 6, July 2020
  36. Muneer Ahmed Salamkar. ETL Vs ELT: A Comprehensive Exploration of Both Methodologies, Including Real-World Applications and Trade-Offs. Distributed Learning and Broad Applications in Scientific Research, vol. 5, Mar. 2019
  37. Naresh Dulam, and Venkataramana Gosukonda. “AI in Healthcare: Big Data and Machine Learning Applications ”. Distributed Learning and Broad Applications in Scientific Research, vol. 5, Aug. 2019
  38. Sarbaree Mishra. A Novel Weight Normalization Technique to Improve Generative Adversarial Network Training. Distributed Learning and Broad Applications in Scientific Research, vol. 5, Sept. 2019

Similar Articles

You may also start an advanced similarity search for this article.