Hadoop Big Data Developer

What we offer

• Remote first and full-time W2 employment with Bright Vision Technologies
• Salary in the range of USD 100,000 to 150,000 per year
• In-house SOW engagements with no third-party clients
• Direct employment with no C2C or 1099 arrangements
• No new H1B sponsorship; H1B transfers supported for qualified candidates
• Collaborative, growth-oriented culture with mentorship and engineering standards
• A mandatory technical coding assessment as part of the hiring process

Role overview

We are seeking a seasoned Hadoop Big Data Developer to design, build, and operate large-scale data processing pipelines and analytics platforms within the Hadoop ecosystem. You will ingest, transform, and analyze vast volumes of structured and unstructured data to support analytics, machine learning, and reporting workloads. The right candidate combines deep technical expertise across Hadoop with solid software engineering practices.

Responsibilities

• Architect, develop, and run end-to-end big data pipelines on Hadoop, pulling data from relational, file-based, streaming, and API sources
• Build robust ETL and ELT workflows using Spark, Hive, Pig, and Sqoop with strong emphasis on data quality, idempotency, error handling, and recoverability
• Develop high-throughput streaming pipelines using Kafka, Spark Streaming, or Flink and integrate with downstream analytics and operational systems
• Tune Spark and MapReduce jobs by adjusting partitioning, memory, serialization, and skew handling to meet demanding SLAs at optimized cost
• Model and organize data on HDFS, Hive, HBase, and modern lakehouse formats (Parquet, ORC, Delta, Iceberg, Hudi) to balance flexibility and performance
• Collaborate on data governance, lineage, and quality controls with governance and security teams
• Implement monitoring, alerting, and logging for pipelines, including job-level SLAs and proactive failure detection
• Partner with data scientists and analysts to deliver curated, reliable datasets with clear documentation
• Automate pipeline orchestration using Airflow, Oozie, or similar tools with clean dependency management and defined ownership
• Continuously evaluate new technologies in big data and cloud ecosystems (EMR, Databricks, Snowflake, BigQuery) for meaningful improvements
• Lead performance reviews and architecture audits, proposing concrete refactoring and optimization initiatives
• Document data architectures, schemas, pipeline behaviors, and runbooks to support scaling teams
• Mentor junior engineers and contribute to engineering standards and best practices

Requirements

• Bachelor’s degree in Computer Science, Engineering, or a related technical field
• Minimum of five years designing and operating big-data pipelines on Hadoop
• Hands-on experience with Apache Spark in production (Scala, Python, or Java)
• Solid experience with Hive, HDFS, Sqoop, HBase, and the broader Hadoop ecosystem
• Practical experience with streaming platforms such as Kafka, Spark Streaming, or Flink
• Strong SQL skills and experience with relational and NoSQL data stores
• Experience with workflow orchestration tools like Airflow or Oozie
• Solid understanding of distributed systems concepts including partitioning, replication, and fault tolerance
• Proficient scripting in Python or Shell
• Excellent troubleshooting, debugging, and documentation abilities

Technologies

• Hadoop, Apache Spark, Scala, Python, Java
• Hive, Pig, Sqoop, Kafka, Spark Streaming, Flink
• HDFS, HBase, Parquet, ORC, Delta, Iceberg, Hudi
• Airflow, Oozie
• EMR, Databricks, Snowflake, BigQuery
• AWS EMR, Azure HDInsight, Kubernetes, Spark-on-K8s
• Trino, Apache Atlas, Collibra, CI/CD, infrastructure-as-code

How to apply

For immediate consideration, please send your resume to [email protected]. Learn more about Bright Vision Technologies at www.bvteck.com. A technical coding assessment is mandatory for all applicants.

Employment terms and visa policy

• This is a 100 percent remote, full-time direct W2 role with Bright Vision Technologies
• In-house Statement of Work engagement; no third-party client involvement
• No C2C or 1099 arrangements
• No new H1B sponsorship is available; H1B transfers are supported for qualified candidates
• A technical coding assessment is required

Equal Employment Opportunity

Bright Vision Technologies is an equal opportunity employer. We do not discriminate based on race, color, religion, sex, sexual orientation, gender identity, national origin, age, disability, veteran status, or any other protected status under applicable laws. We maintain a workplace free of harassment and discrimination.