Wednesday, February 6, 2013

Large Reporting Jobs with Amazon RDS Read Replicas (MySQL)

  I recently came across a project where a monthly report was overwhelming a small multi-AZ MySQL RDS instance. For 29 days of the month the small instance was sufficient, but running the monthly report would not only take days to complete, but it would bog down the entire website while it was running.

  First, a little background. The website in question is a public internet facing website. It consists of two small web server instances behind an Elastic Load Balancer, a small multi-AZ RDS instance, and a small instance used for running administrative processes and reports.

   In a non-cloud environment the only answer would be to either upgrade the database server, or add a second database server dedicated for reporting jobs. Luckily this website was running in AWS.

  With AWS we could easily upgrade the database to a larger instance to support the production load and the monthly report, but the extra capacity provided by the larger instance would be expensive and go unused for all but one day a month. We could manually create a read replica, allowing the reporting job to run against another small instance, but that the job would still take days to complete, and again we'd be paying for unused resources for most of the month. 

  The answer we came up with was simple - we modified the reporting job to take advantage of the AWS API and launch a huge read replica of the database. The job flow went like this:
  1. Launch High Memory Double Extra Large read replica of the RDS instance*
  2. Once the read replica was online execute our monthly report
  3. Destroy read replica once the job completes**
  4. Email report results to interested parties
  This solution is the best of both worlds - The monthly report gets to take advantage of the resources available in the larger database instance to complete in hours instead of days, and the customer only pays for the resources they actually need. Instead of paying several hundred dollars per month to upgrade the multi-AZ instance to support both workloads, the customer ended up paying about $10 a month for better results.

  The moral of the story is that while it's totally possible to treat cloud resources just like traditional servers in a data center, you're usually doing yourself a disservice by doing so. By taking advantage of the extra flexibility and features built into most cloud platforms you can not only save money, but also provide an all around better experience for yourself and your end users.

23 comments:

  1. I admire the valuable information you offer in your articles. I will bookmark your blog and have my friends check up here often. I am quite sure they will learn lots of new stuff here than anybody else! Regards aws jobs in hyderabad.

    ReplyDelete
  2. It's very useful blog post with inforamtive and insightful content and i had good experience with this information. We, at the CRS info solutions ,help candidates in acquiring certificates, master interview questions, and prepare brilliant resumes.Find top Salesforce admin interview questions in 2020.
    These Salesforce developer interview questions are highly helpful in 2020. You can read these Salesforce lightning interview questions and Salesforce integration interview questions which are prepared by industry experts.

    ReplyDelete
  3. Gone through this wonderful coures called Salesforce Certification Training in Dallas who are offering fully practical course, who parent is Salesforce Training in USA and they have students at Salesforce Training classes in Canada institutes.

    ReplyDelete
  4. You can join the free orientation at Workday training institutes in Bangalore and they located at Workday training institutes in delhi. It is highly recommendable for you to enroll at Workday training institutes in UK and you can have a great market for jobs in USA so join this wonderful Workday training institutes in USA and in India this Workday training institutes in Noida is really best institute.

    ReplyDelete
  5. I've also experienced it twice in the software industry when I wasn't enjoying my work, wasn't passionate about it, and didn't believe in the company. In these instances, somebody else liberated me! Salesforce training in Hyderabad

    ReplyDelete
  6. Email Us:- inkdrop121@gmail.com

    We're the producers of legal documents and we provide documents like license, passport, visa, ssn, citizenship certificate and many more. If you are interested or for more information please do not hesitate to get in touch with us via Email Us:- inkdrop121@gmail.com

    Website Link: linkedin.com/in/ink-drop-2a3a39209

    ReplyDelete
  7. You need personal informations from companies,family and friends that will better your life and you need easy access without them noticing just contact james or you’re financially unstable or you have a bad credit score, he will solve that without stress,he and his team can clear criminal records without leaving a trace and can also anonymously credit your empty credit cards with funds you need,all these are not done free obviously but I like working with James and his team cause they keep you updated on every step taken in order to achieve the goal and they also deliver on time,I tested and confirmed this I’m still happy on how my life is improving after my encounter with him ,you can send a mail to premiumhackservices@gmail.com

    ReplyDelete
  8. Simply appreciating your work and thinking about how you dealt with this blog so well. It's striking to such an extent that I can't stand to not experience this important data at whatever point I surf the web! SEO expert Singapore

    ReplyDelete
  9. I am so delighted I found your weblog, I really found you by accident,
    while I was researching on Bing for something else, Regardless I am here now and would just like to say thank you for a fantastic post and a all round entertaining blog (I also love the theme/design), HOW TO GET A FAKE DRIVERS LICENSE THAT WORKS FROM THE DMV I don't have time to go through it all at the minute but I have bookmarked it and also added your RSS feeds, so when I have time I will be back to read a lot more, Please do keep up the excellent job.

    ReplyDelete
  10. Playtech's first ever casino - JRH Hub
    Playtech's first ever casino 전라남도 출장안마 - JRH 하남 출장샵 Hub - 상주 출장마사지 offers more than just online entertainment - the most exciting slots and gaming 군산 출장마사지 machines from the 통영 출장샵 world's top companies.

    ReplyDelete
  11. The Perfect Casino: Top Offers & Bonuses at
    The septcasino Best Casino sol.edu.kg Offers. Casino Bonuses & Promotions. With an 출장샵 emphasis on casino games, 토토사이트 the gambling industry is nba매니아 expected to explode

    ReplyDelete
  12. BITCOIN RECOVERY IS VERY MUCH REAL, AM A LIVING TESTIMONY!!!!

    Have you ever gotten your bitcoins stolen from your wallet or invested in an ICO that turned out to be a scam, you are not alone because this happened to me too. I initially lost $340,000 in just 3 months from Cryptoallday, I contacted the authorities and they referred me to a bitcoin recovery company who helped me recover all my funds within less than 2 days... I`m speaking out to improve our awareness of these cryptocurrency thieves and help you if you have been a victim yourself. Simply send them an email wzardgarryspeedhack@gmail.com
    Whatsapp or text: +1 540 209-1989, website: https://wzardgarryspeedhac.wixsite.com/my-site-1

    ReplyDelete