Senior Data Engineer - PySpark & Data Ingestion

We are seeking a Senior Data Engineer to support our Security Insights platform, designing, implementing, and operating on Palantir. You will be accountable for technical development, analysis, and maintenance of a rolling roadmap for currency, and working with technical teams to ensure execution of the roadmap is delivered as per schedule.

Requirements

• Transforming unstructured datasets into a more consumable format using PySpark / Python
• Ingesting data from upstream source systems through REST API, File Transfer (SFTP), Bash, and SQL, and working with Network Connectivity, Firewall and Proxy in Linux
• Managing data services (ITSM) in an Agile Scrum environment using Jira and ServiceNow, and improving the operations and CI/CD process for the application and team
• Designing and building data pipelines in a distributed computing environment while tuning performance using techniques like pagination, partitioning, and broadcasting
• Taking ownership in engaging with internal Westpac clients based on data governance and management practices (as 2IC or Tech Lead)
• Experience in governance for data onboarding, data quality, master data, and access control (nice to have)
• Experience in the multi-tenancy data lake platforms e.g. Databricks / Snowflake / Cloudera (nice to have)

Benefits

• Competitive remuneration
• Great culture
• Wide range of employee benefits
• Personalised upskilling, mentoring, and training opportunities
• Flexible working
• Generous paid and unpaid parental leave
• Access to super contributions while on parental leave
• Supportive and collaborative team environment
• Opportunities for career growth and development
• Inclusive and diverse workplace