Kuchup
Jobs Interview prep Sign in
Jobs Interview prep Sign in
Jobs · Poland · Capco
CA

Capco

AWS Data Engineer

Posted today

India - Bengaluru, Poland
Full-time
Mid-level
Original posting job-boards.greenhouse.io

Verified from careers page

About the role

Job Title: Data Engineer

About Us

“Capco, a Wipro company, is a global technology and management consulting firm. Awarded with Consultancy of the year in the  British Bank Award and has been ranked Top 100 Best Companies for Women in India 2022 by Avtar & Seramount. With our presence across 32 cities across globe, we support 100+ clients across banking, financial and Energy sectors. We are recognized for our deep transformation execution and delivery. 

WHY JOIN CAPCO? 

You will work on engaging projects with the largest international and local banks, insurance companies, payment service providers and other key players in the industry. The projects that will transform the financial services industry.

MAKE AN IMPACT

Innovative thinking, delivery excellence and thought leadership to help our clients transform their business. Together with our clients and industry partners, we deliver disruptive work that is changing energy and financial services.

#BEYOURSELFATWORK

Capco has a tolerant, open culture that values diversity, inclusivity, and creativity.

CAREER ADVANCEMENT

With no forced hierarchy at Capco, everyone has the opportunity to grow as we grow, taking their career into their own hands.

DIVERSITY & INCLUSION

We believe that diversity of people and perspective gives us a competitive advantage.

Job Description

Role: AWS Data Engineer

Location : Bangalore, Pune

Experience-6-12 yrs 

Databricks Development and Data Engineering

  • Design, develop, test, and maintain scalable data pipelines and transformation workflows using Databricks.
  • Build and optimise data processing solutions using Apache Spark, PySpark, and SQL.
  • Develop batch and, where required, streaming data pipelines to support enterprise data processing requirements.
  • Implement robust data ingestion, cleansing, transformation, and enrichment processes across multiple data sources.
  • Develop and maintain Delta Lake tables, applying appropriate data modelling, partitioning, and optimisation strategies.
  • Implement reusable, modular, and maintainable data engineering frameworks and components.
  • Ensure data pipelines meet agreed performance, reliability, scalability, and quality standards.

Data Platform Design and Optimisation

  • Contribute to the design and implementation of cloud-based data platforms and lakehouse architectures.
  • Apply Medallion Architecture principles, including Bronze, Silver, and Gold data layers, where appropriate.
  • Optimise Spark workloads through efficient transformations, query tuning, cluster configuration, and resource management.
  • Implement appropriate data storage, partitioning, and data lifecycle management strategies.
  • Monitor and troubleshoot data processing workloads, identifying performance bottlenecks and implementing improvements.
  • Support the implementation of secure and scalable data platform solutions aligned with enterprise architecture standards.

Data Integration and Orchestration

  • Design and develop data ingestion and integration pipelines connecting structured, semi-structured, and unstructured data sources.
  • Implement data orchestration workflows using Databricks Workflows, AWS, or equivalent technologies.
  • Support integration with enterprise databases, APIs, cloud storage, and downstream analytical platforms.
  • Develop data validation, reconciliation, and quality control processes to ensure data accuracy and consistency.
  • Implement appropriate error handling, logging, monitoring, and recovery mechanisms.
  • Support data migration and modernisation initiatives, including migration from legacy data platforms to cloud-based solutions.

Software Engineering and DevOps

  • Apply software engineering best practices to data pipeline development, including modular design, code reviews, automated testing, and documentation.
  • Develop and maintain CI/CD pipelines for deploying Databricks code and configurations across environments.
  • Use Git, Azure DevOps or equivalent tools for source control, collaboration, and release management.
  • Implement unit testing, integration testing, and data quality testing for data engineering solutions.
  • Support automated deployment processes using Databricks Asset Bundles, Terraform, or equivalent tools where appropriate.
  • Troubleshoot and resolve defects across development, testing, and production environments.
  • Contribute to continuous improvement of development standards, reusable frameworks, and engineering practices.
  • Data Governance, Security and Reliability
  • Implement appropriate data access controls, security configurations, and governance standards within Databricks.
  • Work with Unity Catalog or equivalent governance capabilities to support data discovery, access management, and lineage.
  • Apply data quality, metadata management, and data lifecycle principles.
  • Support compliance with organisational security policies and relevant financial services regulatory requirements.
  • Implement monitoring and alerting to improve pipeline reliability and operational resilience.
  • Collaborate with platform, security, and architecture teams to ensure solutions meet enterprise standards.

Consulting, Technical Leadership and Stakeholder Engagement

  • Engage with client stakeholders to understand business requirements and translate them into scalable technical solutions.
  • Take ownership of complex technical deliverables or defined data engineering workstreams.
  • Contribute to solution architecture discussions, technical design decisions, and implementation planning.
  • Provide technical guidance, code reviews, and mentoring to junior engineers and consultants.
  • Communicate technical concepts, delivery progress, risks, and dependencies to technical and non-technical stakeholders.
  • Collaborate with cross-functional teams to ensure alignment between data engineering solutions and business objectives.
  • Identify opportunities to improve data engineering practices, platform performance, and delivery efficiency.
  • Support technical estimation, sprint planning, and delivery coordination within Agile teams.

Required Skills and Experience

Essential Technical Skills

  • Typically 6+ years of experience in data engineering, software engineering, or related technology roles, including substantial hands-on experience with Databricks.
  • Strong practical experience developing data pipelines and transformation workloads using Databricks.
  • Advanced proficiency in Python, PySpark, and SQL.
  • Strong understanding of Apache Spark architecture, distributed data processing, and performance optimisation.
  • Hands-on experience with Delta Lake, including data ingestion, transformation, optimisation, and data management.
  • Experience designing and implementing ETL/ELT pipelines for large-scale data processing.
  • Good understanding of lakehouse architecture, data modelling, and data warehousing concepts.
  • Strong knowledge and hands-on experience of AWS development in some of the following : Lambda, Glue, Step Functions, IAM roles, Lakeformation, EventBridge, SNS, SQS, EC2,Security Groups, CloudFormation, RDS, DynamoDB, Redshift
  • Experience with data orchestration tools such as Databricks Workflows or Azure Data Factory.
  • Practical experience implementing data quality checks, error handling, logging, and monitoring.
  • Experience using Git and CI/CD pipelines within a data engineering environment.
  • Understanding of data security, access controls, and governance principles.
  • Strong troubleshooting and performance-tuning capabilities for Spark and Databricks workloads.
  • Experience working within Agile software delivery environments.

Consulting and Professional Skills

  • Strong analytical thinking and structured problem-solving capabilities.
  • Ability to independently own and deliver complex technical workstreams.
  • Experience contributing to technical design and solution architecture decisions.
  • Strong written and verbal communication skills, including the ability to explain technical concepts to non-technical stakeholders.
  • Ability to engage effectively with client stakeholders and multidisciplinary delivery teams.
  • Experience providing technical guidance and mentoring to junior team members.
  • Ability to identify delivery risks, manage dependencies, and recommend appropriate solutions.
  • Strong organisational skills and the ability to manage competing priorities.
  • Proactive, collaborative, and client-focused approach to delivery.
  • Desirable Skills and Experience
  • Experience delivering data engineering solutions within banking, capital markets, insurance, or other financial services environments.
  • Databricks Certified Data Engineer Associate or Professional certification.
  • Experience with Unity Catalog, Delta Live Tables / Lakeflow Spark Declarative Pipelines, and Databricks SQL.
  • Exposure to structured streaming and real-time data processing using Spark Structured Streaming or Kafka.
  • Experience with infrastructure as code using Terraform or equivalent technologies.
  • Familiarity with Microsoft Fabric, Snowflake, Synapse Analytics, or other modern data platforms.
  • Experience with cloud data migration and legacy data platform modernisation programmes.
  • Knowledge of dimensional modelling, Data Vault, and enterprise data architecture principles.
  • Familiarity with data observability, metadata management, and data lineage solutions.
  • Exposure to cost optimisation and workload management within cloud data platforms.
  • Understanding of data governance and regulatory considerations relevant to financial services.
  • Experience supporting technical proof-of-concepts and evaluating emerging data technologies.

 

  • Excellent verbal communication skills.
  • Good interpersonal skills.
  • Ability to provide technical leadership for the team.

    

 

 

Prepare for this interview Questions tailored to this role with clear answers Start prep →
Apply with Kuchup Save job
Original posting job-boards.greenhouse.io

Verified from careers page

Apply with Kuchup

Kuchup

Jobs straight from company careers pages

Jobs Interview prep About Privacy