The DSE is responsible for preparing "big data” infrastructure to include the design, build, and integration of various data sources. The Contractor shall perform the following tasks:
- Develop complex queries to ensure accessibility while optimizing the performance of NoSQL and or big data infrastructure. Build and optimize big data data pipelines, architectures and data sets.
- Build and maintain the infrastructure to support ETL processing. Extract data from multiple data sources, such as SQL, MongoDB, and other platform APIs, and load into a centralized data warehouse to facilitate unified reporting.
- Configure and manage data analytic frameworks, databases and tools such as Spark, Hadoop, Kafka, Hive, Pig, NoSQL, SQL, HDInsight, MongoDB, Cassandra and graph databases like Neo4j, GraphDB, and OrientDB.
- Apply distributed systems concepts and principles such as consistency and availability, liveness and safety, durability, reliability, fault-tolerance, consensus algorithms.
- Administrate cloud computing and CI CD pipelines to include Azure, AWS and Hanscom milCloud.
Location: MacDill, AFB