Software Craftsman & DevOp - SEO
Overview
As an experienced software craftsman and DevOp, I successfully completed a project that involved developing a CI/CD pipeline using Jenkins and Docker, extracting data using Spark and Hadoop, and implementing automated acceptance testing. With my deep understanding of software development methodologies and modern technologies, I was able to design and implement an HTML parser with XPath that allowed effective structuring of 750 million data per country. I also extended and rebuilt an HTML crawler that enabled efficient data extraction using Spark and Hadoop for 200 million seeds. I also built a PaaS using Apache Mesos, Marathon, and AWS Route 53, and created automated acceptance tests for an existing SaaS tool. Finally, I created a status dashboard using Play to provide visibility into the project’s status.
My “You Build It, You Run It” methodology means that I am responsible not only for building the software, but also for ensuring that the implementation is effective and runs smoothly.
Activities
- ”You Build It, You Run It” CI/CD pipeline build with Jenkins and Docker.
- Requirements analysis, conception, design, implementation and operation of an HTML parser using XPath.
- Extract structured data ( 750 million per country) and make it available as an API
- Extend/build an html crawler for data extraction with Spark and Hadoop for 200 million seeds
- PaaS build using Apache Mesos, Marathon, and AWS Route 53-Create automated acceptance tests for an existing SaaS tool.
- Status dashboards with Play
Methodology
- ”You Build It, You Run It”
Tools / Technologies
- Java
- RxJava
- Play Framework
- Scala
- Groovy
- Javascript
- HTML
- CSS
- Spark
- Hadoop
- Nutch
- Tika,Swagger
- SBT
- SAXON-HE
- Memcached
- Ehcache
- Infinispan
- MySQL 5.7.x
- JOOQ
- AnrangoDB
- Bash,CSV
- Guice
- Mockito
- Cucumber
- Jenkins
- Git
- Docker
- Jira
- Confluence
- Markdown