Spark streaming reduceByKeyAndWindow unstable application

I wanted to run a job which runs 24×7 and which reports if certain keywords occur more than a N times in the stream. Spark streaming looked a ideal candidate for this task. Spark has a reduceByKeyAndWindow function which was exactly what I was looking for. I decided to use a window length of 1 minute and […]

Spark streaming: Fixing all executors not getting jobs

I was working on a feature recently which needed a streaming job that runs 24×7 and processing 100 million rows per day. The spark web ui is a wonderful tool to look at how things are running internally. While debugging I noticed that the streaming jobs were getting allocated to only one machine. Spark has […]

Sending mails from munin using custom command

We use Sendgrid for email delivery on our cloud. Using custom commands in munin can be tricky. After struggling for some time I wrote a small python script to send emails from munin using sendgrid. Add this line to munin.conf contacts mycontact contact.mycontact.command /usr/bin/sendemail alerts@example.com “[Munin] Alert for ${var:host}” “warnings: ${var:wfields} / criticals: ${var:cfields}” contact.mycontact.text text […]

Motorcycle review: Bajaj Avenger 2013

I bought Avenger in July 2013. Specification can be read at http://www.bajajauto.com/avenger/. I have toured extensively in Karnataka and Leh, so I think this review will definitely help you make up your mind. General: Engine: The 220 cc engine gets you off from 0-60 pretty quickly. This is a cruiser so couple of seconds does not […]

Winter Ride

2014 was an incredible year for me. I travelled a lot. I decided to end this year with another ride with some fellow bikers. This was a 4-day ride in which I covered Mudigere, Kudremukh, Sringeri, Agumbe, Chikmanglur and Mullayangiri. We rode on the first day to Mudigere. A home stay was arranged for stay. It […]

Setting up VPN using OpenVPN on Google Cloud or AWS

VPN is a good way to protect your cloud infrastructure from attacks. In the method below I will list down the step how to log in to your vpn machine and access the cloud LAN from you laptop. I am using Debian on Google Cloud as an example. The setup process should be similar on […]

Tips for planning your motorcyle trip to Leh

Planning: Read and research. Start here: http://www.bcmtouring.com/travel-gui…-travel-guide/ http://www.team-bhp.com/forum/route-…ate-guide.html See lot of photos, will keep you excited 🙂 You can ride solo, however I would highly recommend against it. Form a group on Bangalore Bikers or BMC Touring. Ride with experienced bikers. Newbies will find it daunting to ride everyday. I had a guy in my group […]

Day 13-14 – Beautiful Jispa – Back to Manali

We decided to take the day off and enjoy this beautiful village. Jispa is a very small village with couple of hotels and tents. Green trees, a river and snow filled mountain tops. We decided to head to the river. Yak? I’ll come back! The trip is going to be over 🙁 River Chandra and […]

Day 12 – Ice bucket challenge! – Pang – Jispa

We woke up next day to freezing cold at Pang. One of the guys went to take a dump. After he was done, he noticed that water in the cans were frozen. Ice bucket challenge accepted 🙂 I leave the rest to your imagination. I will never ever complain about road conditions! Vishal Taal Ice […]