Power Of Open Source Software

Hey, I'm a Software Developer who uses Open Source Software often. But finally it took me three years to witness the freaking power of Open Source Software because of the POC task which i was assigned recently.


So, What does a POC task mean?

        POC means ‘Proof of Concept’. In Software development, if the team weren't sure of any Open Source Software that will suit the requirement, the task will be bucketed to POC. Here the developer to whom it was assigned will be working on it and based on the end result, the team will decide whether the same idea can be implemented further.

Hope you got a quick overview on what a POC is. 

Let me share my experience in short,

To Do -

        We had to copy our ElasticSearch data to Snowflake DB for analytics stuff. But then, there were no direct ways to establish a pipeline to send the data to Snowflake. So we were hunting for intermediaries and finally we decided to copy this through Amazon S3. 

Yes you are right , the flow will be like

            ElasticSearch -> AWS S3 -> Snowflake

As simple as that.
Here, my POC scope is to get ElasticSearch data as JSON in AWS S3 bucket.

The blast starts here:

The First day,

    I was quite excited as this was my first POC task. I used to work on developing the UI’s and enhancing the backends. In past, I worked on developing new/existing features with existing tech stacks, but haven’t worked on anything from scratch. The first day, I was exploring and analysing about docker containers, ruby rails etc.., as we decided to copy ElasticSearch data to AWS S3 via Logstash and some plugins.

    After some hours of exploring, I was all set to create a docker container for Logstash. Once the docker is up, it seemed to be like a cakewalk. Yes, If the values are correct for all the configuration file entries and roles of AWS is properly set, the Docker container gets created in no time. But still I was hunting for help from my seniors, Stack Overflow and many forums to find the configuration entries. The first day ended without any progress because I couldn’t find what’s wrong with the configuration files.

The Next day,

    I was completely demotivated because the docker creation wasn’t successful. After getting help from one of my seniors, I was able to figure out the correct IAM role for S3 which  helped me to create the Docker container. This kick started my motivation. Later this day, I ended installing all the plugins required for this task and was able to copy the ElasticSearch data to S3. And there comes the next hurdle, the data in S3 looked like an object.

The Day,

    I was desperately searching for the plugins to see json data in AWS S3. Then I found this ‘Codec-JSON’ plugin which made me to type semicolon to my POC task and start my weekend vibes. I was happy when I saw the data in json format in S3.


Everything that was running in my mind during this task was,

    

What if logstash docker and the plugins that I hunted for, were not created or made open source?

    Obviously, My team need to hire 10 more developers and have to work on this for month. But thankfully we have all this Docker images & plugins which made our work easier and saved a lot of our time. People/Companies who share their work and make it as an open source looked like a hero to me after this POC task. In my college days, I had FOSS paper for semester, but it never left a deep impact of Open source as this POC task did.

Yeah, it’s true..!!

If the concept of Open Source doesn’t exist, then we would be somewhere 5-10 years behind in the Software Development era.

With lots of gratitude,

End of this post…