decor
 

planetDB2 logo

Planet DB2 is an aggregator of blogs about the IBM DB2 database server. We combine and republish posts by bloggers around the world. Email us to have your blog included.

 

September 27, 2016


DB2 Guys

IBM DB2 – the database for the cognitive era at IBM World of Watson 2016

IBM Insight, the premiere data, analytics and cognitive IBM conference, is now part of IBM World of Watson 2016 to be held at Las Vegas from October 24-27.  This year attendees will be able to experience first-hand a world of cognitive capabilities that IBM has been at the forefront of. World of Watson incorporates the […]

Triton Consulting

DB2 12 Latest News – Join the IBM Webcast featuring Jeff Josten and Julian Stuhler

We are delighted to let you know that Julian Stuhler, Solutions Delivery Director at Triton Consulting and IBM Gold Consultant will be joining the panel on IBM’s next DB2 12 webcast on Tuesday 4th October. Register here Julian will be speaking … Continue reading →

(Read more)

DB2utor

DB2 12 In-Memory Index Optimization

Last month I wrote about the trend of increasing the amount of real storage when configuring new mainframe systems. DB2 12 (which is set for delivery in the fourth quarter of 2016) features many new performance improvements that are designed to take advantage of available real storage. One of these enhancements is called In-Memory Index Optimization -- aka, index fast traversal blocks (FTBs).
 

September 26, 2016

Big Data University

Introducing Two New SystemT Information Extraction Courses

This article on information extraction is authored by Laura Chiticariu and Yunyao Li.

We are all hungry to extract more insight from data. Unfortunately, most of the world’s data is not stored in neat rows and columns. Much of the world’s information is hidden in plain sight in text. As humans, we can read and understand the text. The challenge is to teach machines how to understand text and further draw insights from the wealth of information present in text. This problem is known as Text Analytics.

An important component of Text Analytics is Information Extraction. Information extraction (IE) refers to the task of extracting structured information from unstructured or semi-structured machine-readable documents. It has been a well-known task in the Natural Language Processing (NLP) community for a few decades.

Two New Information Extraction Courses

We just released two courses on Big Data University that get you up and running with Information Extraction in no time.

The first one, Text Analytics – Getting Results with System T introduces the field of Information Extraction and how to use a specific system, SystemT, to solve your Information Extraction problem. At the end of this class, you will know how to write your own extractor using the SystemT visual development environment.

The second one, Advanced Text Analytics – Getting Results with System T goes into details about the SystemT optimizer and how it addresses the limitations of previous IE technologies. For a brief introduction to how SystemT will solve your Information Extraction problems, read on.

Common Applications of Information Extraction

The recent rise of Big Data analytics has led to reignited interest in IE, a foundational technology for a wide range of emerging enterprise applications. Here are a few examples.

Financial Analytics. For regulatory compliance, companies submit periodic reports about their quarterly and yearly accounting and financial metrics to regulatory authorities such as the Securities and Exchange Committee. Unfortunately, the reports are in textual format, with most of the data reported in tables with complex structures. In order to automate the task of analyzing the financial health of companies and whether they comply with regulations, Information Extraction is used extract the relevant financial metrics from the textual reports and make them available in structured form to downstream analytics.

Data-Driven Customer Relationship Management (CRM).  The ubiquity of user-created content, particularly those on social media, has opened up new possibilities for a wide range of CRM applications. IE over such content, in combination with internal enterprise data (such as product catalogs and customer call logs), enables enterprises to have a deep understanding of their customers to an extent never possible before.Besides demographic information of their individual customers, IE can extract important information from user-created content and allows enterprises to build detailed profiles for their customers, such as their opinions towards a brand/product/service, their product interests (e.g. “Buying a new car tomorrow!” indicating the intent to buy car), and their travel plans (“Looking forward to our vacation in Hawaii” implies intent to travel) among many other things.  Such comprehensive customer profiles allow the enterprise to manage customer relationship tailored to different demographics at

Besides demographic information of their individual customers, IE can extract important information from user-created content and allows enterprises to build detailed profiles for their customers, such as their opinions towards a brand/product/service, their product interests (e.g. “Buying a new car tomorrow!” indicating the intent to buy car), and their travel plans (“Looking forward to our vacation in Hawaii” implies intent to travel) among many other things. Such comprehensive customer profiles allow the enterprise to manage customer relationship tailored to different demographics at

Such comprehensive customer profiles allow the enterprise to manage customer relationship tailored to different demographics at fine granularity, and even to individual customers. For example, a credit card company can offer special incentives to customers who have indicated plans to travel abroad in the near future and encourage them to use credit cards offered by the company while overseas.

Machine Data Analytics. Modern production facilities consist of many computerized machines performing specialized tasks. All these machines produce a constant stream of system log data. Using IE over the machine-generated log data it is possible to automatically extract individual pieces of information from each log record and piece them together into information about individual production sessions. Such session information permits advanced analytics over machine data such as root cause analysis and machine failure prediction.

A Brief Introduction to SystemT

SystemT is a state-of-the-art Information Extraction system. SystemT allows to express a variety of algorithms for performing information extraction, and automatically optimizes them for efficient runtime execution. SystemT started as a research project in IBM Research – Almaden in 2006 and is now commercially available as IBM BigInsights Text Analytics.

On the high level, SystemT consists of the following three major parts:

1. Language for expressing NLP algorithms. The AQL (Annotation Query Language) language is a declarative language that provides powerful primitives needed in IE tasks including:

  • Morphological Processing including tokenization, part of speech detection, and finding matches of dictionaries of terms;
  • Other Core primitives such as finding matches of regular expressions, performing span operations (e.g., checking if a span is followed by another span) and relational operations (unioning, subtracting, filtering sets of extraction results);
  • Semantic Role Labeling primitives providing information at the level of each sentence, of who did what to whom, where and in what manner;
  • Machine Learning Primitives to embed a machine learning algorithm for training and scoring.

2. Development Environment. The development environment provides facilities for users to construct and refine information extraction programs (i.e., extractors). The development environment supports two kinds of users:

  • Data scientists who do may not wish to learn how to code can develop their extractor in a visual drag-and-drop environment loaded with a variety of prebuilt extractors that they can adapt for a new domain and build on top of. The visual extractor is converted behind the scenes into AQL code.

Information Extraction

  • NLP engineers can write extractors directly using AQL. An example simple statement in AQL is shown below. The language itself looks a lot like SQL, the language for querying relational databases. The familiarity of many software developers with SQL helps them in learning and using AQL.

AQL Information Extraction

3. Optimizer and Runtime Environment. AQL is a declarative language: the developer declares the semantics of the extractor in AQL in a logical way, without specifying how the AQL program should be executed. During compilation, the SystemT Optimizer analyzes the AQL program and breaks it down into specialized individual operations that are necessary to produce the output.

The Optimizer then enumerates many different plans, or ways in which individual operators can be combined together to compute the output, estimates the cost of these plans, and chooses one plan that looks most efficient.

This process is very similar to how SQL queries are optimized in relational database systems, but the types of optimizations are geared towards text operations which are CPU-intensive, as opposed to I/O intensive operations as in relational databases. This helps the productivity of the developer since they only need to focus on “what” to extract, and leave the question of the “how” to do it efficiently to be figured out by the Optimizer.

Given a compiled extractor, the Runtime Environment instantiates and executes the corresponding physical operators. The runtime engine is highly optimized and memory efficient, allowing it to be easily embedded inside the processing pipeline of a larger application. The Runtime has a document-a-time executive model: It receives a continuous stream of documents, annotates each document and output the annotations for further application-specific processing. The source of the document stream depends on the overall applications.

Advantages of SystemT

SystemT handles gracefully requirements dictated by modern applications such as the ones described above. Specifically:

  • Scalability. The SystemT Optimizer and Runtime engine ensures the high-performance execution of the extractors over individual documents. In our tests with many different scenarios, SystemT extractors run extremely fast on a variety of documents, ranging from very small documents such as Twitter messages of 140 bytes to very large documents of tens of megabytes.
  • Expressivity. AQL enables developers to write extractors in a compact manner, and provides a rich set of primitives to handle both natural language text (in many different languages) as well as other kinds of text such as machine generated data, or tables. A few AQL statements may be able to express complex extraction semantics that may require hundreds or thousands lines of code. Furthermore, one can implement functionalities not yet available via AQL natively via User Defined Functions (UDFs). For instance, developers can leverage AQL to extract complex features for statistical machine learning algorithms, and in turn embed the learned models back into AQL.
  • Transparency. As a declarative language, AQL allows developers to focus on what to extract rather than how to extract when developing extractors. It enables developers to write extractors in a much more compact manner, with better readability and maintainability. Since all operations are declared explicitly, it is possible to trace a particular result and understand exactly why and how it is produced, and thus to correct a mistake at its source. Thus, AQL extractors are easy to comprehend, debug and adapt to a new domain.

If you’d like to learn more about how SystemT handles these requirements and how to create your own extractors, enroll today in Text Analytics – Getting Results with System T and then Advanced Text Analytics – Getting Results with System T.

The post Introducing Two New SystemT Information Extraction Courses appeared first on Big Data University.

 

September 23, 2016


Robert Catterall

DB2 for z/OS: Using PGFIX(YES) Buffer Pools? Don't Forget About Large Page Frames

Not long ago, I was reviewing an organization's production DB2 for z/OS environment, and I saw something I very much like to see: a REALLY BIG buffer pool configuration. In fact, it was the biggest buffer pool configuration I'd ever seen for a single DB2 subsystem: 162 GB (that's the combined size of all the buffer pools allocated for the subsystem). Is that irresponsibly large -- so large as to negatively impact other work in the system by putting undue pressure on the z/OS LPAR's central...

(Read more)

DB2Night Replays

The DB2Night Show #184: DB2 - The Corner Stone of IBM Analytics

Follow @LesKing00 !function(d,s,id){var js,fjs=d.getElementsByTagName(s)[0],p=/^http:/.test(d.location)?'http':'https';if(!d.getElementById(id)){js=d.createElement(s);js.id=id;js.src=p+'://platform.twitter.com/widgets.js';fjs.parentNode.insertBefore(js,fjs);}}(document, 'script', 'twitter-wjs'); Special Guest: Les King, Director of Big Things, IBM DB2 - The Corner Stone of the IBM Analytics Platform Strategy 100% of our audience learned...

(Read more)
 

September 22, 2016


Kim May

TFG Website and Blog – We’re Back!

Many thanks to our colleagues at Substance151, particularly Ida Cheinman, for their rapid response to a website error blocking access to the many documents and presentations attached to both our...

(Read more)
 

September 21, 2016

Jack Vamvas

HOW TO clear inactive DB2 LUW transaction log files

How to clear inactive DB2 LUW transaction log files is a solution to a common problem. Before we discuss how to prune the inactive transaction logs we’ll need to establish which log files are inactive.

It is important to identify which logs to delete. If you delete an active transaction log file, you will cause an outage on the database

Before you start any activity , speak to the owners and users of the database . It is safer to stop the applications. It is not always possible - as it may be a live online database. Also consider create a backup for recovery . Completing a DB2 online backup won't force any applications

Follow the steps below to clear the DB2 LUW transaction log files.

>>Connect to the database using the command

su - db2usr1

$db2 connect to myDB

>>Connect to the database configuration details using either of these methods

$db2 get db config

db2 get db config | grep "log”

>>Search for the First active log file database parameter. For example:

First active log file = S0130573.LOG

>>Once you define the active log file execute

$db2 prune logfile prior to <activeLogFileName>

>>Applying this example to our command:

$db2 prune logfile prior to S0130573.LOG

 >>On execution of this command, DB2 would clear all inactive transaction logs prior to S0130573.LOG

 

Read More

DB2 – Restore database from a ONLINE backup (DBA DB2)

DB2 SQL2413N - Online backup is not allowed (DBA DB2)

 

 

September 20, 2016

Big Data University

This Week in Data Science (September 20, 2016)

Here’s this week’s news in Data Science and Big Data. AI robot

Don’t forget to subscribe if you find this useful!

Interesting Data Science Articles and News

Upcoming Data Science Events

New in Big Data University

  • Data Science Fundamentals Learning Path – When a butterfly flaps its wings what happens? Does it fly away and move on to another flower or is there a spike in the rotation of wind turbines in the British Isles. Come be exposed to the world of data science where we are working to create order out of chaos that will blow you away!

The post This Week in Data Science (September 20, 2016) appeared first on Big Data University.


DB2utor

Open Source Tools and Languages for z/OS

The application development landscape on the mainframe -- which for me is really z/OS -- continues to evolve. Now businesses that are moving to cloud and mobile applications use not only Java, but also Perl, PHP, Python, R and TCL. These open source languages are very powerful because they can facilitate certain types of processing through a minimal amount of coding.
 

September 14, 2016


ChannelDB2 Videos

DB2 Tips n Tricks Part 95 - How To Protect Backup Image using DB2 Native Encryption


Thumbnail

How To Protect Backup Image using DB2 Native Encryption implement DB2 Encryption Feature Happy Learning & Sharing

Dave Beulke

Process to Justify an IBM DB2 Analytics Accelerator (IDAA) Part 1

Unbelievably, the IBM DB2 Analytics Accelerator (IDAA) has been available for many years, helping all types of customers improve overall processing, especially their analytical processing. Many companies do not have an IDAA appliance helping their DB2 for z/OS environments. Since most every shop’s...

(Read more)
 

September 13, 2016


Craig Mullins

The Tao of DB2 - Part 7: Dealing with Performance Issues

The last time we checked in on our DBAs (the soon-to-retire mentor and his intern) the mentor was schooling his young intern on storage and data retention issues. But there is still much to learn, as our intern will soon find out! "Soon you will get the chance to learn about performance tuning," said the mentor, as he nodded solemnly in his chair. As if on cue, one of the programmers came...

(Read more)

DB2utor

Become a Part of Generation z

In my job with IBM, I spend a lot of time at customer sites. During these visits, I’m seeing an increasing number of young IT professionals who are just starting their careers on z/OS.
Big Data University

This Week in Data Science (September 13, 2016)

Here’s this week’s news in Data Science and Big Data. NBA data

Don’t forget to subscribe if you find this useful!

Interesting Data Science Articles and News

Upcoming Data Science Events

Cool New Courses

The post This Week in Data Science (September 13, 2016) appeared first on Big Data University.

 

September 12, 2016


Data and Technology

A Dozen SQL Rules of Thumb, Part 3

Today we pick up our three-part series of SQL rules of thumb (ROTs) with the third and final installment… You can think of these rules as general guiding principles you should follow as your...

(Read more)
 

September 06, 2016

Big Data University

This Week in Data Science (September 06, 2016)

Here’s this week’s news in Data Science and Big Data. Tech Ethics

Don’t forget to subscribe if you find this useful!

Interesting Data Science Articles and News

Upcoming Data Science Events

The post This Week in Data Science (September 06, 2016) appeared first on Big Data University.


DB2utor

DB2 12 Continuous Delivery Model Webinar

Software development on z/OS has traditionally adhered to tried and true practices to ensure that poorly constructed application code does not make it into production. In many shops we call these change control people the “gatekeepers.” Nothing gets past these individuals. Developers must rigorously test their code before an application or update goes live. Beyond that, fallback procedures must be devised in case there's a problem once the code is moved into production. On top of that, changes to production systems are subject to a strict schedule to ensure that no critical business event are interrupted.
 

September 03, 2016


DB2Night Replays

The DB2Night Show #183: DB2 LUW Security from the Data Center to the Cloud!

Follow @Roger_E_Sanders !function(d,s,id){var js,fjs=d.getElementsByTagName(s)[0],p=/^http:/.test(d.location)?'http':'https';if(!d.getElementById(id)){js=d.createElement(s);js.id=id;js.src=p+'://platform.twitter.com/widgets.js';fjs.parentNode.insertBefore(js,fjs);}}(document, 'script', 'twitter-wjs'); Special Guest: Roger Sanders, Author, Teacher, and Security Guru, IBM DB2 LUW Security: From the Data Center to the Cloud! 100% of our...

(Read more)
 

August 30, 2016

Jack Vamvas

Troubleshoot Long running sql statements with LONG_RUNNING_SQL view

A customer called me yesterday and complained about slow response time on a DB2 database. I asked a few questions and they mentioned a few  ad-hoc queries were executing.  If I’m doing production troubleshooting – running a sql trace is a powerful method to retrieve detailed information.

Prior to getting the detailed information , I’ll use the SYSIBMADM.LONG_RUNNING_SQL view . This is a very useful DB2 administrative view , presenting long – running queries.

Under the hood , the LONG_RUNNING_SQL view joins some system snapshots.

 

 SELECT APPL_NAME,AUTHID,INBOUND_COMM_ADDRESS, STMT_TEXT, AGENT_ID,   ELAPSED_TIME_MIN, APPL_STATUS, DBPARTITIONNUM   FROM SYSIBMADM.LONG_RUNNING_SQL ORDER BY APPL_NAME

 

Check the APPL_STATUS value on the query. Some troubleshooting scenarios which I use are:

a) Finding LOCKWAIT – Use the lock snapshots to dig deeper into the source of the issue

b) Find UOWAIT – check the requesting application

Read More

How to read db2detaileventlock event monitor trace file (DBA DB2)

Database Tuning for complex sql queries (DBA DB2)

Database Tuning – Five Basic Principles according to Shasha (DBA ...

Big Data University

This Week in Data Science (August 30, 2016)

Here’s this week’s news in Data Science and Big Data. Self Driving Car

Don’t forget to subscribe if you find this useful!

Interesting Data Science Articles and News

Upcoming Data Science Events

The post This Week in Data Science (August 30, 2016) appeared first on Big Data University.


DB2utor

Greater Memory Already Making a Great Impact

In November 2015 I wrote about the launch of DB2 12 for z/OS ESP, citing all the various enhancements DB2 has made by exploiting significantly greater amounts of available memory. Well, it isn't just DB2 12 that will benefit from additional memory to reduce CPU cost.

Robert Catterall

DB2 for z/OS: Clearing Up Some Matters Pertaining to Database Access Threads

I have recently received a number of questions pertaining to DB2 for z/OS database access threads, or DBATs. DBATs are threads used in the execution of SQL statements that are sent to DB2 from network-attached applications (i.e., from DRDA requesters that access DB2 for z/OS by way of DB2's distributed data facility, also known as DDF). Thinking that these questions (and associated answers) might be of interest to a good many people in the DB2 for z/OS community, I'm packaging them in this blog...

(Read more)
 

August 26, 2016


ChannelDB2 Videos

DB2 Tips n Tricks Part 94 - How To Find Tablespaces included inside Tablespace Level Backup Image


Thumbnail

How To Find Tablespaces included inside Tablespace Level Backup Image db2ckbkp -T imgname Happy Learning & Sharing

ChannelDB2 Videos

DB2 Tips n Tricks Part 93 - How LOGARCHMETH2 is not alternative for LOGARCHMETH1


Thumbnail

How LOGARCHMETH2 is not alternative or backup for LOGARCHMETH1 Configure failarchlog DB CFG Parameter Happy Learning & Sharing
Jack Vamvas

How to write a DB2 loop with INSERT

Question: I’d like write a sql statement to loop through an INSERT statement and increment with a  count. The purpose is to create some test tables for load testing.

Answer: It is possible to create a loop in DB2 which loops through an incremental INSERT. This is a basic example, which can be customised for your purposes.Note the use of ATOMIC. The purpose of ATOMIC is to rollback before the call is passed back to the requestor, if there is a problem.

In this example the CNT variable increments at every INSERT, up until it is under 100000.

 

db2 “CREATE TABLE mytble (ID INT)”
db2 "BEGIN ATOMIC DECLARE CNT INT DEFAULT 5; WHILE CNT < 100000 DO INSERT INTO mytbl (ID) VALUES('16'); SET CNT = CNT + 1; END WHILE;END"

 Read More

Software unit testing and DB2 sql loop test code (DBA DB2)

DB2 Tuning Toolkit – DB2 Design advisor - Ddb2advis

 

 

 

 

 

 

 

 

August 25, 2016


Data and Technology

A Dozen SQL Rules of Thumb, Part 2

Today’s blog post picks up where we left off in our three-part series of rules of thumb (ROTs) that apply generally to SQL development regardless of the underlying DBMS. These are the general guiding...

(Read more)
 

August 24, 2016


Dave Beulke

3 Consideration for Enjoying the Data Lake

With all the outside activities with friends and family, summer vacations are always wonderful. Being outside at the lake enjoying the warm weather and cooling off in the lake are a wonderfully relaxing great times. This is the safe, content image that everyone thinks about when discussing the new...

(Read more)
 

August 23, 2016

Big Data University

This Week in Data Science (August 23, 2016)

Here’s this week’s news in Data Science and Big Data. Connected Volcano

Don’t forget to subscribe if you find this useful!

Interesting Data Science Articles and News

Upcoming Data Science Events

  • Constant Contact: Using IBM BigInsights to Create Business Insight – Join this session on August 25th to learn how Constant Contact, a leader in email marketing, is using IBM BigInsights to create useful insights for their clients in a way that scales.
  • IBM i2 Summit – Join the IBM i2 Summit on August 30-31 to hear directly from experts who are using all forms of data, including “dark data,” to outthink threats.
  • Combining IBM SPSS Statistics and R for competitive advantage – This Data Science Central Webinar event on September 1st, will show you how SPSS Statistics can help you keep up with the influx of new data and make faster, better business decisions without coding.
  • Big Data and Health presented by IBM Canada – Join We Are Wearables Toronto and IBM Canada on September 16th for a look at how wearables and sensors are changing healthcare.
  • How Data Can Help in the Fight Against Sexual Assault – Join the Center for Data Innovation and Rise, a civil rights nonprofit, on October 6th in Washington D.C., for a panel discussion on how policymakers and law enforcement can develop data-driven policies and practices to help in the fight against sexual assault and improve the lives of survivors.

The post This Week in Data Science (August 23, 2016) appeared first on Big Data University.


DB2utor

Setting Special Registers Using Profiles

DB2 11 introduced a powerful feature that allows you to set special registers for remote applications by creating a profile in the profile tables. Information on the install jobs used to create these tables, as well as rules for inserting rows into the tables, can be found in the DB2 Administration Guide or the IBM Knowledge Center.
 

August 22, 2016


Craig Mullins

The Tao of DB2 - Part 6: That Storage Stuff

When we last looked in on our DBAs (the soon-to-retire mentor and his intern) the intern was learning about backup and recovery and had put together a comprehensive backup plan for the new system when it would move to production.  As the intern reclined in his chair viewing the magnificent backup plan with pride, his mentor glanced at it and admitted that the intern had done well. Just...

(Read more)

Henrik Loeser

Notes on Notebooks, Data, DB2, and Bluemix

Weather Graph in Jupyter Notebook Some time ago I shared with you how I used open data like weather data and CeBIT statistics to have some fun with Jupyter Notebooks. Notebooks are an old, but now -...

(Read more)

Subscribe by email

 

About

planetDB2 is an aggregator of blogs about the IBM DB2 database server. We combine and republish posts by bloggers around the world. Email us to have your blog included.
 

Bloggers

decor