Data Engineering

Job Description

Company Overview:

Our organization is a leading innovator in cybersecurity, cloud, and AI solutions, dedicated to developing cutting-edge products and services that address the evolving needs of the technology landscape. We thrive in a rapidly developing market (Indonesia) where the demand for advanced tech solutions is ever-growing, driven by rapid technological advancements. We are an AI-native company committed to continuous improvement, helping our customers unlock their full revenue potential.

Role Summary

As a Data Engineer you will play a crucial role in building and managing the data pipelines that are essential for training and fine-tuning our Large Language Models (LLMs), with a specific focus on the Indonesian language. You will be responsible for designing, building, and maintaining a robust and scalable data infrastructure. You will collaborate closely with our team of Data Scientists and Machine Learning Engineers to ensure the availability of high-quality, clean, and structured Indonesian language data for developing accurate and locally relevant AI models.

Key Responsibilities

• Build and Manage Data Pipelines: Design, develop, and maintain ETL (Extract, Transform, Load) processes to collect and process Indonesian text data from various sources, such as databases, APIs, and log files.
• Data Collection and Integration: Gather complex and relevant datasets tailored to business needs, particularly Indonesian text data that covers a wide range of dialects and linguistic styles.
• Data Cleaning and Pre-processing: Perform data cleaning to handle inconsistent, duplicate, or corrupted data. You will also transform raw data into a usable format for training machine learning models.
• Data Architecture: Design and implement an efficient and scalable data architecture, including data warehouses and data lakes, to store and manage large volumes of data.
• Ensure Data Quality: Develop data validation methods and analysis tools to ensure the integrity and accuracy of the data used for model training.
• Team Collaboration: Work closely with Data Scientists to understand their data requirements and provide ready-to-use data for the fine-tuning and evaluation of LLM models.
• Performance Optimization: Monitor and optimize the performance of data pipelines to ensure efficiency and scalability, especially when handling very large volumes of data.
  
  

Requirements

Qualifications & Experience:

Education

• Required: Bachelor’s degree in Computer Science, Engineering, Information Technology, or a related quantitative field.
• Preferred: Master’s degree in a Computer Science.
  
  

Experience

• Required: Minimum 3 to 5 years of hands-on experience in a data engineering role, particularly in projects involving big data and machine learning with a proven track record of designing and implementing data pipelines and architecture including data ingestion, storage, processing and delivery.
• Preferred: Experience with Big Data technologies (e.g., Hadoop, Spark). Experience with cloud platforms (e.g., AWS, GCP, Azure) and their associated data services. Familiarity with DevOps/DataOps principles for CI/CD.
  
  

Required Skills

• Technical Skills:
• Programming Languages: High proficiency in programming languages such as Python, SQL, and Scala.
• Databases: Deep understanding of relational databases (like MySQL, PostgreSQL) and NoSQL databases (like MongoDB).
• Big Data Tools: Hands-on experience with big data technologies such as Apache Spark, Hadoop, and Kafka.
• Cloud Computing: Knowledge of cloud platforms like Amazon Web Services (AWS), Google Cloud Platform (GCP), or Microsoft Azure.
• ETL Tools: Familiarity with ETL tools like Apache Airflow, Talend, or Stitch.
• Soft Skills:
• Strong analytical and problem-solving abilities.
• Excellent communication and teamwork skills to collaborate effectively with various teams.
• Ability to work independently in a dynamic environment.
  
  

Competencies

• Technical:
• Architecture Design
• Business Needs Analysis
• Data Analysis and Interpretation
• Infrastructure Design
• Software Design
• Solution Architecture
• System Architecture Design
• System Configuration Management
• System Integration
• Leadership:
• Applied Learning
• Building Customer Loyalty
• Business Awareness
• Collaborating
• Continuous Improvement
• Planning & Organizing
• Quality Orientation