decor
 

planetDB2 logo

Planet DB2 is an aggregator of blogs about the IBM DB2 database server. We combine and republish posts by bloggers around the world. Email us to have your blog included.

 

July 16, 2018


Craig Mullins

Broadcom Set to Purchase CA Technologies for Close to $19 Billion

If you've been paying attention the past couple of days you no doubt will have heard that Broadcom, heretofore known for their semiconductors, has made a bid to acquire CA Technologies. I've been expecting something to happen ever since the rumors of a merger with BMC were rampant about a year ago. Broadcom is offering an all cash deal for CA, but many analysts and customers of both...

(Read more)

Keith McDonald

The Haddon Matrix

In the embedded video, we examine the Haddon Matrix, a framework from the injury prevention literature. We then apply it to how a software team prevents and deals with an outage.

The Haddon Matrix

To minimize the risk of bad outcomes, experts in injury-prevention rely on a framework called the Haddon Matrix. This framework provides a way to think about accidents by allowing you to focus on three periods of time: pre-event, event, and post-event.

The 3-dimensional Haddon Matrix showing pre-event, event, and post-event on the phases dimension

The Haddon Matrix is also useful for thinking about preventing, handling, and recovering from other types of situations that aren’t necessarily life-threatening. For example, a software team that owns a web service may want to think through the kinds of problems that could take their service off-line and what they’d do about it.

Pre-Event

Think about the pre-event. What are some reasons the site could have gone down and what could we have done to prevent them?

  • The team may have a pushed a bad code change.
  • The physical hardware their service was running on failed.
  • The system became overloaded with unanticipated demand.

We could prevent most bad code changes through:

  • code review or pairing
  • automated tests
  • automated deployment (CI/CD)
  • monitoring with automatic rollback

We could prevent physical hardware failures from affecting customers through redundancy. Or, we could have a second host in a different region with an automatic fast failover mechanism.

We could avoid failing under greater load by buying more hardware resources than we need, or better yet, through the use of elastic cloud computing resources. Also, we could also load test our software before we release it.

The Event

Think about the event. What could we do in the moment to minimize the damage? In the case of a site outage due to bad code change, we could make it easy to do a manual rollback. For hardware failure, as in the prevention case, we could have a second system to manually fail-over to in the moment. In the face of demand exceeding capacity, having a way to scale up resources on demand would help.

In all three cases, having monitoring and logging that helps you find the cause of the outage is critical. So is having runlists or standard operating procedures for dealing with these kinds of outages in a calm manner that doesn’t put things in an even worse state. We would also need a proper on call rotation with well-trained operations people.

Post-Event

Think about the post-event. This may be about having a mechanism in place to ensure it doesn’t repeat, such as having an event post-mortem as part of your process and taking action on the lessons that come out of the post-mortem. Thinking about the post-event scenarios may also help focus on the cost to the company of losing customer trust and potentially having to compensate customers for the downtime. An honest assessment of the costs and risks may help with prioritizing some event and pre-event activities like investing in redundant hardware, or monitoring, or a continuous delivery system.

Benefits of The Haddon Matrix

The beauty of the Haddon Matrix is that it helps you think of environmental solutions rather than relying exclusively on carrots and sticks. In this exercise, we created a robust plan against site outages without thinking about the team’s Riders and Elephants. So, we simply tweaked the environment to help make the negative outcome less likely.

I would love to hear what you thought of the video, so feel free to comment below, on The K Guy Twitter, or on The K Guy Facebook fan page.

The post The Haddon Matrix appeared first on The K Guy.


Henrik Loeser

Extended: Manage and interact with Watson Assistant from the command line

Remember my blog posts about how to manage Watson Assistant from the command line and how to test context for a conversation? Well, that tool did not work well for server actions which I used in this...

(Read more)

Data and Technology

My Computer Mug Collection, Part 3

So far I have published two posts showing the computer-related mugs in my collection.  Over several weeks I first tweeted a mug a day from my collection, and now I am blogging them for those that...

(Read more)
 

July 13, 2018


Henrik Loeser

How to: Enable Let's Encrypt SSL wildcard certificates for your IBM Cloud apps

Private data - secured Early last year, I wrote about how to improve security for your custom domains on IBM Cloud. I pointed out that I was using the bluemix-letsencrypt wrapper for Cloud Foundry...

(Read more)
Jack Vamvas

How to list Linux non-system local users

Question: I'd like to capture Linux local users who are not system users and also return the server name and group associated with the local user. What statement can I execute to create this list ?

 

 Answer: Linux offers a number of different methods to report on local users and other meta data associated with the user such as groups and permissions .

This code snippet will return non-system users and exclude "user1" and "user2". I've added that predicate to give you some flexibility around how you construct the code.

Based on the details  - the hostname and the group of the user is returned

Run the code through cmdline 

  Linux_local_users

Read more on Linux

3 signs of low Linux memory (DBA DB2)

Powershell grep and awk (DBA DB2)

 

July 10, 2018


Data and Technology

My Computer Mug Collection, Part 2

Last week I posted part 1 of my computer mug collection and promised to continue posting the rest of the collection in later blog posts. Well, here is the second in this series of posts. Today we...

(Read more)
 

July 05, 2018


Data and Technology

My Computer Mug Collection, Part 1

Recently I have been tweeting a mug a day from my collection of coffeee mugs given to me over the years by computer and software companies. I just tweeted the last one (I think), and since these...

(Read more)
 

July 03, 2018


Henrik Loeser

Learn about chatbots at upcoming IBM Cloud Meetup

Learn how to chat at the Meetup Want to quickly and easily build a chatbot, integrate it with Slack, Facebook Messenger or other platforms? Connect the bot with a database? Join me at the IBM Cloud...

(Read more)
 

June 29, 2018


Triton Consulting

Less than 5 months to go until Black Friday 2018 – have you started to plan?

According to research by PWC the vast majority of Black Friday spending will be done online. This means that retailers are under immense pressure to ensure that their IT systems are up to the job. We’ve come up with a … Continue reading → The post Less than 5 months to go until Black...

(Read more)
 

June 28, 2018


Kim May

IBM Think Conference Call for Speakers is **OPEN**

The IBM Think 2019 Conference will be held in San Francisco, on February 12 – 15.  For those like me, who felt the Las Vegas conferences were becoming one exhausted blur, San Francisco is a welcome...

(Read more)
 

June 26, 2018


Robert Catterall

Db2 for z/OS PBG Table Spaces: The Ins and Outs of Partition Addition

In an entry posted to this blog about three years ago, I advocated for putting some thought into the PBG versus PBR decision (partition-by-growth versus partition-by-range) for larger tables (for smaller tables, PBG is often the sensible choice, especially when the associated table space is likely to have only one partition). I pointed out in the aforementioned blog entry several advantages pertaining to PBR table spaces in a large-table context, but acknowledged as well the attractiveness...

(Read more)

Henrik Loeser

Enable Botkit Middleware for Watson Assistant for serverless actions

Slack chatbot with Watson Assistant Last week I blogged about using Botkit Middleware to create a Watson-powered database interface. I pointed to a tutorial demonstrating a Slack chatbot for a SQL...

(Read more)
 

June 25, 2018


Data and Technology

Don’t Forget the Free Space

When you design your databases you must take care to specify appropriate free space to allow for data growth. Free space, sometimes called fill factor, is a DDL option that can be specified when...

(Read more)
 

June 22, 2018

Big Data University

Read and Write CSV Files in Python Directly From the Cloud

Every data scientist I know spends a lot of time handling data that originates in CSV files. You can quickly end up with a mess of CSV files located in your Documents, Downloads, Desktop, and other random folders on your hard drive.

I greatly simplified my workflow the moment I started organizing all my CSV files in my Cloud account. Now I always know where my files are and I can read them directly from the Cloud using JupyterLab (the new Jupyter UI) or my Python scripts.

This article will teach you how to read your CSV files hosted on the Cloud in Python as well as how to write files to that same Cloud account.

I’ll use IBM Cloud Object Storage, an affordable, reliable, and secure Cloud storage solution. (Since I work at IBM, I’ll also let you in on a secret of how to get 10 Terabytes for a whole year, entirely for free.) This article will help you get started with IBM Cloud Object Storage and make the most of this offer. It is composed of three parts:

  1. How to use IBM Cloud Object Storage to store your files;
  2. Reading CSV files in Python from Object Storage;
  3. Writing CSV files to Object Storage (also in Python of course).

The best way to follow along with this article is to go through the accompanying Jupyter notebook either on Cognitive Class Labs (our free JupyterLab Cloud environment) or downloading the notebook from GitHub and running it yourself. If you opt for Cognitive Class Labs, once you sign in, you will able to select the IBM Cloud Object Storage Tutorial as shown in the image below.

IBM Cloud Object Storage Tutorial

 

What is Object Storage and why should you use it?

The “Storage” part of object storage is pretty straightforward, but what exactly is an object and why would you want to store one? An object is basically any conceivable data. It could be a text file, a song, or a picture. For the purposes of this tutorial, our objects will all be CSV files.

Unlike a typical filesystem (like the one used by the device you’re reading this article on) where files are grouped in hierarchies of directories/folders, object storage has a flat structure. All objects are stored in groups called buckets. This structure allows for better performance, massive scalability, and cost-effectiveness.

By the end of this article, you will know how to store your files on IBM Cloud Object Storage and easily access them using Python.

 

Provisioning an Object Storage Instance on IBM Cloud

Sign up or log in with your IBM Cloud account here (it’s free) to begin provisioning your Object Storage instance. Feel free to use the Lite plan, which is free and allows you to store up to 25 GB per month. You can customize the Service Name if you wish, or just leave it as the default. You can also leave the resource group to the default. Resource groups are useful to organize your resources on IBM Cloud, particularly when you have many of them running. When you’re ready, click the Create button to finish provisioning your Object Storage instance.

Creating an Object Storage instance

Working with Buckets

Since you just created the instance, you’ll now be presented with options to create a bucket. You can always find your Object Storage instance by selecting it from your IBM Cloud Dashboard.

There’s a limit of 100 buckets per Object Storage instance, but each bucket can hold billions of objects. In practice, how many buckets you need will be dictated by your availability and resilience needs.

For the purposes of this tutorial, a single bucket will do just fine.

Creating your First Bucket

Click the Create Bucket button and you’ll be shown a window like the one below, where you can customize some details of your Bucket. All these options may seem overwhelming at the moment, but don’t worry, we’ll explain them in a moment. They are part of what makes this service so customizable, should you have the need later on.

Creating an Object Storage bucket

If you don’t care about the nuances of bucket configuration, you can type in any unique name you like and press the Create button, leaving all other options to their defaults. You can then skip to the Putting Objects in Buckets section below. If you would like to learn about what these options mean, read on.

Configuring your bucket

Resiliency Options

Resiliency OptionDescription
Characteristics
Cross RegionYour data is stored across three geographic regions within your selected locationHigh availability and very high durability
RegionalYour data is stored across three different data centers within a single geographic regionHigh availability and durability, very low latency for regional users
Single Data CenterYour data is stored across multiple devices within a single data centerData locality

Storage Class Options

Frequency of Data AccessIBM Cloud Object Storage Class
ContinualStandard
Weekly or monthlyVault
Less than once a monthCold Vault
UnpredictableFlex

Feel free to experiment with different configurations, but I recommend choosing “Standard” for your storage class for this tutorial’s purposes. Any resilience option will do.

After you’ve created your bucket, store the name of the bucket into the Python variable below (replace cc-tutorial with the name of your bucket) either in your Jupyter notebook or a Python script.

Creating Service Credentials

To access your IBM Cloud Object Storage instance from anywhere other than the web interface, you will need to create credentials. Click the New credential button under the Service credentials section to get started.

In the next window, select Manager as your role, and add {"HMAC":true} to the Add Inline Configuration Parameters (Optional) field. You can leave all other fields as their defaults and click the Add button to continue.

You’ll now be able to click on View credentials to obtain the JSON object containing the credentials you just created. You’ll want to store everything you see in a credentials variable like the one below (obviously, replace the placeholder values with your own). Take special note of your access_key_id and secret_access_key which you will need for the Cyberduck section below.

Note: If you’re following along within a notebook be careful not to share this notebook after adding your credentials!

 

Putting Objects in Buckets

There are many ways to add objects to your bucket, but we’ll start by taking a look at two simple ways: the IBM Cloud web interface and Cyberduck.

IBM Cloud Web Interface

You can add a CSV file of your choice to your newly created bucket through the web interface by either clicking the Add objects button, or dragging and dropping your CSV file into the IBM Cloud window.

If you don’t have an interesting CSV file handy, I recommend downloading FiveThirtyEight’s 2018 World Cup predictions.

Cyberduck

Cyberduck is a free cloud storage browser for Mac OS and Windows. It allows you to easily manage all of the files in all of your object storage instances. After downloading, installing, and starting Cyberduck, create a new bookmark by pressing +Shift+B on Mac OS or Ctrl+Shift+B on Windows. A window will pop up with some bookmark configuration options. Select the Amazon S3 option from the dropdown and fill in the form as follows:

  • Nickname: enter anything you like.
  • Server: enter your service endpoint. You can choose any public endpoint here. For your convenience, I recommend one of these:
    • s3-api.us-geo.objectstorage.softlayer.net (If you live in the Americas)
    • s3.eu-geo.objectstorage.softlayer.net (if you live in Europe)
    • s3.ap-geo.objectstorage.softlayer.net (if you live in Asia)
  • Access Key ID: enter the access_key_id you created above in the Creating Service Credentials section.

Close the window and double-click on your newly created bookmark. You will be asked to log in. Enter the secret_access_key_id you created above in the Creating Service Credentials section and click Login.

You should now see a file browser pane with the bucket you created in the Working with Buckets section. If you added a file in the previous step, you should also be able to expand your bucket to view the file. Using the action dropdown or the context menu (right-click on Windows, control-click on Mac OS).

You can add files to your buckets by dragging and dropping them onto this window.

Whether you use the IBM Cloud web interface or Cyberduck, assign the name of the CSV file you upload to the variable filename below so that you can easily refer to it later.

 

Reading CSV files from Object Storage with Cyberduck

Once you have successfully accessed an object storage instance in Cyberduck using the above steps, you can download files by double-clicking them in Cyberduck’s file browser. You can also generate links to your files by selecting the Open/Copy Link URL option.

By default your files are not publicly accessible, so selecting a URL that is not pre-signed will not allow the file to be downloaded. Pre-signed URLs do allow for files to be downloaded, but the link will eventually expire. If you want a permanently available public link to one of your files, you can select the Info option for that file and add READ permissions for Everyone under the permissions section.

 

After changing this setting you can share the URL (without pre-signing) and anyone with the link will be able to download it, either by opening the link in their web browser, or by using a tool like wget from within your Jupyter notebook, e.g.

Reading CSV files from Object Storage using Python

The recommended way to access IBM Cloud Object Storage with Python is to use the ibm_boto3 library, which we’ll import below.

The primary way to interact with IBM Cloud Object Storage through ibm_boto3 is by using an ibm_boto3.resource object. This resource-based interface abstracts away the low-level REST interface between you and your Object Storage instance.

Run the cell below to create a resource Python object using the IBM Cloud Object Storage credentials you filled in above.

After creating a resource object, we can easily access any of our Cloud objects by specifying a bucket name and a key (in our case the key is a filename) to our resource.Object method and calling the get method on the result. In order to get the object into a useful format, we’ll do some processing to turn it into a pandas dataframe.

 

We’ll make this into a function so we can easily use it later:

Adding files to IBM Cloud Object Storage with Python

IBM Cloud Object Storage’s web interface makes it easy to add new objects to your buckets, but at some point you will probably want to handle creating objects through Python programmatically. The put_object method allows you to do this.

In order to use it you will need:

  1. The name of the bucket you want to add the object to;
  2. A unique name (Key) for the new object;
  3. A bytes-like object, which you can get from:
    • urllib‘s request.urlopen(...).read() method, e.g.
      urllib.request.urlopen('https://example.com/my-csv-file.csv').read()
    • Python’s built-in open method in binary mode, e.g.
      open('myfile.csv', 'rb')

To demonstrate, let’s add another CSV file to our bucket. This time we’ll use FiveThirtyEight’s airline safety dataset.

You can now easily access your newly created object using the function we defined above in the Reading from Object Storage using Python section.

Get 10 Terabytes of IBM Cloud Object Storage for free

You now know how to read from and write to IBM Cloud Object Storage using Python! Well done. The ability to pragmatically read and write files to the Cloud will be quite handy when working from scripts and Jupyter notebooks.

If you build applications or do data science, we also have a great offer for you. You can apply to become an IBM Partner at no cost to you and receive 10 Terabytes of space to play and build applications with.

You can sign up by simply filling the embedded form below. If you are unable to fill the form, you can click here to open the form in a new window.

Just make sure that you apply with a business email (even your own domain name if you are a freelancer) as free email accounts like Gmail, Hotmail, and Yahoo are automatically rejected.


DB2Night Replays

The DB2Night Show #204: Db2 LUW success story with SAP on AWS

Follow @sap_aws Special Guest: Arwel Owen, Enterprise Systems Manager, Princess LTD, UK Db2 LUW Success Story with SAP on AWS 83% of our audience learned something! 17% of our audience were Arwel's co-workers! Arwel completed a VERY successful migration from Oracle to Db2, and then another migration to Db2 V11.1.2.2 SAP on AWS Cloud. The compression, cost savings, and performance results are simply ASTOUNDING! Watch our fun and...

(Read more)
 

June 20, 2018


Triton Consulting

Five go to IDUG

It’s a full house! We’re delighted to announce that all five of Triton’s IDUG team have had their presentations accepted, four as primary speakers and one as a co-speaker. Here’s what you have got to look forward to: Award winning … Continue reading → The post Five go to IDUG appeared...

(Read more)

Craig Mullins

Fast and Effective Db2 for z/OS Test Data Management with BCV5

Perhaps the most significant requirement for coding successful Db2 application programs is having a reasonable set of test data to use during the development process. Without data, there is no way to test your code to make sure it runs. But there are many issues that must be overcome in order to have a useful test data management process. Today we will talk about this along with a key enabling...

(Read more)

Triton Consulting

Dr, Dr, my data is corrupt!

Five signs that you need a Remote DB2 support solution! We speak to our very own in-house Dr about the signs to look out for which can tell you that it’s time to start thinking about a remote DB2 support … Continue reading → The post Dr, Dr, my data is corrupt! appeared first on Triton...

(Read more)
 

June 19, 2018


Henrik Loeser

DeDUG-Treffen in Ehningen / Db2 User Group meeting near Stuttgart

Db2 User Group Meeting Remember the last German Db2 User Group (DeDUG) meetings at IBM in Munich (April) and at Fiducia GAD in Karlsruhe (January)? Or not? Anyway, get ready for an educative meeting...

(Read more)
 

June 18, 2018


Henrik Loeser

Use BotKit Middleware to create Watson-powered database interface

Chatbot anyone? Few months back, I introduced you to a tutorial for a database-drive Slackbot. In the tutorial I used the Conversation connector to hook up IBM Watson Assistant to Slack as a...

(Read more)
 

June 15, 2018


Craig Mullins

Db2 for z/OS Performance Traces Part 3 - Trace Destinations and IFCIDs

In parts 1 and 2 of this series we examined each of the six types of Db2 traces that you can start to get Db2 to record information about its activities. Be sure to check out those earlier blog posts to learn about what is available.  Today we are going to look at a couple of additional details with regard to Db2 traces, starting with where the trace records can be written. Trace...

(Read more)

DB2Night Replays

Db2Night Show #z88: Db2 Security Best Practices Volume II

Presented by: Dave Beulke, IBM Gold Consultant and Champion for IM "The Db2Night Show z88: Db2 Security Best Practices Volume II" Replays available in WMV and M4V formats! 100% of our studio audience learned something!Dave discussed the latest Db2 security methods, procedures and techniques for fortifying Db2 systems, databases and applications. Watch the replay...

(Read more)
 

June 14, 2018


Kim May

*FREE* Hands-on IDAA Training – Columbus Class Announced for July

Gold Consultant and Z Systems Champion Frank Fillmore will deliver a free, hands-on IBM Db2 Analytics Accelerator/IDAA class on Monday and Tuesday, July 23rd and 24th, at the IBM Dublin Campus near...

(Read more)
 

June 13, 2018


Henrik Loeser

Securing your Python app with OpenID Connect (OIDC)

Authorized access only using OpenID Connect Some weeks back I introduced to a tutorial on how to analyse GitHub traffic. The tutorial combines serverless technology and Cloud Foundry to...

(Read more)

Craig Mullins

Db2 for z/OS Performance Traces Part 2 - Global, Monitor, Performance, and Statistics

In Part 1 of the series on Db2 performance traces we provided a general overview, as well as a discussion of the Accounting and Audit trace classes. Today, in Part 2, we will discuss the remaining 4 trace classes: Global, Monitor, Performance, and Statistics. Global Trace The global trace is one I hope you never have to use. It produces information that is used to service Db2, so you'd...

(Read more)
 

June 12, 2018

 

June 08, 2018


Craig Mullins

Db2 for z/OS Performance Traces Part 1 - Intro, Accounting, and Audit

When it comes to Db2 for z/OS performance management, one of the first things you need to understand is the traces that are available that must be started in order to cause Db2 to track performance metrics. You can think of a Db2 trace as a window into the performance characteristics of aspects of your workload. Db2 traces record diagnostic information describing specific performance events. As...

(Read more)

DB2Night Replays

The DB2Night Show #206: A crash course on Db2 pureScale and Advantages

Follow @mohankumarsp Follow @rcollins963 Special Guests: Kent Collins and Mohan Saraswatipura, Authors A crash course on Db2 pureScale and Advantages 100% of our audience learned something! Kent provided a wealth of excellent information on Db2 pureScale including best practices, database manager configuration settings, database configuration settings, registry variables, HADR considerations, how to size memory, monitoring, workload...

(Read more)
 

June 06, 2018


Triton Consulting

Cloudy with a Chance of DevOps – A Survival Guide Webcast

We are delighted to announce that Julian Stuhler, Solutions Delivery Director at Triton Consulting has been invited to present the first in a series of IBM webcasts by IBM Gold Consultants. Julian will be taking a look at DevOps for … Continue reading → The post Cloudy with a Chance of...

(Read more)

Henrik Loeser

Tutorial: Analyze and visualize open data with Apache Spark

Life Expectancy Map Many government agencies and public administrations offer access to data, contributing to open data. Using IBM Watson Studio with Jupyter Notebooks and Apache Spark it is...

(Read more)

Triton Consulting

What kind of summer break will your DBA be having? Sun, sea and sangria or a hotel lobby, dodgy WIFI and support calls?

We’re finally thawing out after a particularly long winter. The #beastfromtheeast is a distant memory and thoughts are turning to blue skies and beaches. Now is the time to start planning for that tricky holiday silly-season. If your organisation has … Continue reading → The post What...

(Read more)

Subscribe by email

 

About

planetDB2 is an aggregator of blogs about the IBM DB2 database server. We combine and republish posts by bloggers around the world. Email us to have your blog included.
 

Bloggers

decor