Lead Software Infra Engineer  

Responsibilities:

• Work closely with the product development teams and engineers to understand their priorities and their technical problems to solve and working together with the other team leads on strategies and planning.
• Work on new components as well as legacy systems.
• Work in core systems, CICD pipeline tools, and observability & alerting tools.
• Work on the production environment as well as development environments.
• Lead infra engineers to deliver improvements in infrastructure and tools to:
  • Allow all engineers to develop, test, and deploy software products more rapidly and consistently (DevOps).
  • Ensure they are stable and scalable with growing traffic and sufficiently monitored (SRE).
  • Allow all engineers to troubleshoot and analyze production issues quickly (SRE)
• For each project assigned to your engineers,
  • do project planning and be actively involved in the technical design and analysis, manage and update the progress, and be accountable for every delivery.
  • Actively involved in all stages of the development process for all technical projects solving engineers’ obstacles to ensure timely and quality delivery.
  • Actively involved in hands-on implementation whenever needed (about 20-40% of your time) such as occasionally writing the initial codebase so other engineers can follow and build upon it.
  • Identify things that could go wrong before engineers start implementing but also willing to be hands-on with the team investigating ugly production bugs.
• For each infra component and tool owned by your team
  • Ensure security controls are being planned and implemented.
  • Ensure monitoring and alerting are put in place.
• For incidents or vulnerabilities reported
  • Manage and participate in responding and resolving infrastructure and tool issues reported in development as well as in production.
  • Manage and participate in on-call activities in and outside of working hours to respond, investigate for RCA, and escalate real issues in production.
  • Ensure RCA is always done, is clearly documented and lessons learned applied.
• For the team,
  • Ensure all engineers apply and tailor software development concepts and best practices.
  • Do regular 1-1s with the other engineers to ensure their work is aligned with business goals and still motivating to them.
  • For each engineer, define technical areas and concrete goals for them to achieve and keep track of the progress, and give support as needed.
  • Understand the capabilities of each engineer, mentor them as effective developers, and allow them to raise their technical expertise.
  • Help with hiring: help screen profiles, take part in interviews, review case study, and share feedback and recommendations.

Skill Requirements:

• 5-10 years of experience designing, building, and scaling software cloud infrastructure and CICD and monitoring tools for large and high throughput public facing web services.
• 3-5 years of experience leading and growing technical teams to deliver values.
• Great team player, have excellent work ethic, self organized, and can manage time well.
• Possess excellent problem solving skills, ability to break down problems and explain complex concepts, strong attention to detail.
• Have deep knowledge and long working experience in:
  • Both the DevOps and SRE culture, practices, and skills.
  • Linux based OS, working in shell, and with open source tools.
  • Data storages and databases, especially PostgreSQL.
  • Tooling for monitoring and alerting, especially Grafana, Prometheus, Datadog, and EFK.
  • Infrastructure-as-code, such as Ansible, Kubernetes, Terraform, and plain Shell scripting.
  • Complex networking and API gateways, especially Kong and Nginx.
  • In-memory databases like RabbitMQ, Redis, and Kafka.
  • Cloud platforms and cloud native architecture
• Have excellent written communications skills to be able to articulate the detailed information among team members and also externally to non-technical groups.
• Able to communicate in English both verbally and in writing.