Showing posts with label db2. Show all posts
Showing posts with label db2. Show all posts

Fetching Lines From Big Files

Today I had a DB2 bulk loading problem. I got a couple of rejected records on a 2G+ file. These kind of files are too big to be manipulated with conventional text editors, and even text editors that can handle such files are, usually, not very fast.
Since I was in Windows, I've open th MSYS shell, changed into the correct directory and typed
cat -n STPRRIMH.S.ULD.U110115.UN | grep 1943664 > MH.txt
This is an instantiation of a more generic command
cat -n < file > | grep < line >
In a few seconds I had a new file containing exactly the information I needed.

Serious developers can't really live without Unix/Linux tools on their machines...

./M6

Interpreting DB2 JDBC error messages

Every time I use DB2 on a data migration project I get awkward DB2/JDBC errors, no matter if I'm using DB2 on Windows, Unix or iSeries AS/400.
Part of the problem is that the error messages come in Portuguese, making the debug task a lot harder since it's almost impossible to find decent help by searching through the error messages. The other part of the problem is that this leaves me the SQL error codes, which sometimes are unclear and makes me waste time.
Here's an example of a common problem that I usually have:
com.ibm.db2.jcc.b.rg: [jcc][t4][102][10040][3.50.152] Non-atomic batch failure. The batch was submitted, but at least one exception occurred 
on an individual member of the batch.
Use getNextException() to retrieve the exceptions for specific batched elements.

com.ibm.db2.jcc.b.pm: Error for batch element #1: The current
transaction was rolled back because of error "-289".. SQLCODE=-1476,
SQLSTATE=40506, DRIVER=3.50.152

com.ibm.db2.jcc.b.SqlException: [jcc][103][10843][3.50.152]
[...] ERRORCODE=-4225, SQLSTATE=null
Following DB2 official documentation:
  • SQLCODE=-1476 means that the current transaction was rolled back because of error.
  • SQLSTATE=40506 means that the current transaction was rolled back because of an SQL error, which is basically the same as the SQLCODE above.
  • ERRORCODE=-4225 means an error occurred when data was sent to a server or received from a server, which is totally useless.
The real useful information is hidden in the second error message, in the 'transaction was rolled back because of error "-289"' message. The key here is the -289 error.
This error means "Unable to allocate new pages in table space", and this is the real cause for such a big fuss.

One of my DB2 table spaces run out of space and all I get is a lot of fuss about a rollback that happened because of an error but the error itself is kind of hidden in the middle of the stack trace, all that is shown is a loose error code...
IBM has done it again, the cause of the error should be highlighted and perfectly visible and understood in order to know what really happened and fix the problem, but making the life easier for its users seems not to be IBM way.

./M6

Execute Commands Inside DB2 Procedures

I was looking for a way to call DB2 runstats, a non-SQL command, from inside a store procedure.
It turns out that it is actually quite easy. All one has to do is to use the ADMIN_CMD procedure from the SYSPROC schema.

ADMIN_CMD is used by applications to run DB2 command line processor administrative commands using the SQL CALL statement.

Here's how to do it: ADMIN_CMD

./M6

DB2 How To

While I was searching for the loading method that accepted a dynamic query from inside a stored procedure, I found a simple and very useful DB2 How To.

Here it is: DB2 How Tos.

./M6

DB2 Copy Schema With Tables

Recently I needed to automate a DB2 schema backup.
My requirements were quite easy, just create a new schema and copy the tables and their content into the new schema.

What initially started as a simple task reveled to be a hard task and resulted in a quite simple script.

The first problem was to execute several commands in the DB2 Command Editor. I was getting syntax errors because the statement termination character was set to be the same as the SQL end statement character ;.
Setting it to # solved my problem, but I did waste some time with this so keep in mind to change the statement termination character to avoid such problem.

I've created a specific schema for this automation task COPY_DATABASE_SCHEMA, so all the procedures will be created there.

Here is the procedure that will copy a table content:

/**
* Copies, with replacement, the content from the source table
* into the target table.
* It will not check for success, i.e. no loading validation
* will be performed.
*
* @sourceTable: the full qualified source table name
* @targetTable: the full qualified target table name
*/
create procedure COPY_DATABASE_SCHEMA.COPY_TABLE(
sourceTable VARCHAR(128),
targetTable VARCHAR(128))
LANGUAGE SQL
begin
declare v_version_number INTEGER default 1;
declare v_cursor_statement VARCHAR(32672);
declare v_load_command VARCHAR(32672);
declare v_sqlcode INTEGER default -1;
declare v_sqlmessage VARCHAR(2048) default '';
declare v_rows_read BIGINT default -1 ;
declare v_rows_skipped BIGINT default -1;
declare v_rows_loaded BIGINT default -1;
declare v_rows_rejected BIGINT default -1;
declare v_rows_deleted BIGINT default -1;
declare v_rows_committed BIGINT default -1;
declare v_rows_part_read BIGINT default -1;
declare v_rows_part_rejected BIGINT default -1;
declare v_rows_part_partitioned BIGINT default -1;
declare v_mpp_load_summary VARCHAR(32672) default NULL;

set v_cursor_statement =
'DECLARE C1 CURSOR FOR SELECT * from ' || sourceTable;
set v_load_command =
'load from C1 of cursor insert into ' || targetTable;

call db2load(1, v_cursor_statement, v_load_command, v_sqlcode,
v_sqlmessage, v_rows_read, v_rows_skipped,
v_rows_loaded, v_rows_rejected, v_rows_deleted,
v_rows_committed, v_rows_part_read,
v_rows_part_rejected, v_rows_part_partitioned,
v_mpp_load_summary) ;
end#
Please note that the target table must already be created in order for this procedure to work.
This procedure was the hard part. Copying the contents of any table requires a dynamic prepared query statement to use as a cursor, and the load command is not available inside a procedure, so the db2load procedure from SYSPROC must be used.

Here is the procedure that will create a schema, create the tables on it and copy the contents:


/**
* Copies an entire schema into a new schema named with the current date.
* @sourceSchema: the source schema name
* @targetSchema: the target schema name
* @tableNameSelection: the tables name to include ('%' for all tables)
*/
CREATE PROCEDURE COPY_DATABASE_SCHEMA.COPY_DATABASE(
sourceSchema VARCHAR(50),
targetSchema VARCHAR(50),
tableNameSelection VARCHAR(150)
)
LANGUAGE SQL
BEGIN
-- Variables
DECLARE stmtSchema VARCHAR(250);
DECLARE stmtTableStructure VARCHAR(200);
DECLARE stmtTableContents VARCHAR(250);
DECLARE stmtAlias VARCHAR(250);
DECLARE tableName VARCHAR(128);
DECLARE numPages BIGINT;
DECLARE nError INTEGER DEFAULT 0;
DECLARE at_end SMALLINT DEFAULT 0;
DECLARE not_found CONDITION FOR SQLSTATE '02000';
DECLARE V_SQL VARCHAR(200);
DECLARE V_STMT STATEMENT;
DECLARE V_LOAD_STMT STATEMENT;
DECLARE TGT_TABLE_CUR CURSOR WITH HOLD WITH RETURN FOR V_STMT;
DECLARE LOAD_CUR CURSOR FOR V_LOAD_STMT;
DECLARE CONTINUE HANDLER for not_found SET at_end = 1;


-- Create schema
SET stmtSchema = char('CREATE SCHEMA ' concat char(targetSchema));
EXECUTE IMMEDIATE stmtSchema;
SET CURRENT SCHEMA targetSchema;


-- Copy tables and views
SET V_SQL = 'SELECT name, npages FROM SYSIBM.SYSTABLES
WHERE CREATOR = ''' || trim(sourceSchema) || '''
AND NAME LIKE ''' || trim(tableNameSelection) || '''
order by name';
PREPARE V_STMT FROM V_SQL;
OPEN TGT_TABLE_CUR;

fetch_loop:
LOOP
FETCH TGT_TABLE_CUR INTO tableName, numPages;
IF at_end <> 0 THEN
LEAVE fetch_loop;
ELSE
SET stmtTableStructure = 'CREATE TABLE ' ||
targetSchema || '.' || tableName || ' LIKE ' ||
sourceSchema || '.' || tableName ;
EXECUTE IMMEDIATE stmtTableStructure;

IF numPages > 0 THEN
call COPY_DATABASE_SCHEMA.copy_table(
sourceSchema || '.' || tableName,
targetSchema || '.' || tableName);
END IF;
END IF;
END LOOP fetch_loop;

CLOSE TGT_TABLE_CUR;
END#
It dynamically issues DB2 SQL commands to create the schema and the tables structure and makes use of the previously created COPY_TABLE to copy the tables content. See the code comments for more information.

To use this just
call COPY_DATABASE_SCHEMA.COPY_DATABASE('originalSchemaName',
'newBackupSchemaName', '%');


Since I also need to discard old copies, I have created a procedure to drop a schema, including all the tables on it.
/**
* Drops an entire schema even if it has tables.
* Views are not supported.
* @schemaName: the name of the schema to drop
*/
CREATE PROCEDURE COPY_DATABASE_SCHEMA.DROP_SCHEMA(
schemaName VARCHAR(50)
)
LANGUAGE SQL
BEGIN
-- Variables
DECLARE stmtDropSchema VARCHAR(250);
DECLARE stmtDropTable VARCHAR(250);
DECLARE tableName VARCHAR(128);
DECLARE at_end SMALLINT DEFAULT 0;
DECLARE not_found CONDITION FOR SQLSTATE '02000';
DECLARE V_SQL VARCHAR(200);
DECLARE V_STMT STATEMENT;
DECLARE TGT_TABLE_CUR CURSOR WITH RETURN FOR V_STMT;
DECLARE CONTINUE HANDLER for not_found SET at_end = 1;

-- Copy tables and views
SET V_SQL = 'SELECT name FROM SYSIBM.SYSTABLES
WHERE CREATOR = ''' || trim(schemaName) || '''
AND CREATOR NOT LIKE ''SYS%''';
PREPARE V_STMT FROM V_SQL;
OPEN TGT_TABLE_CUR;

fetch_loop:
LOOP
FETCH TGT_TABLE_CUR INTO tableName;
IF at_end <> 0 THEN LEAVE fetch_loop;
ELSE
SET stmtDropTable = 'DROP TABLE ' ||
schemaName || '.' || tableName;
EXECUTE IMMEDIATE stmtDropTable;
END IF;
END LOOP fetch_loop;

CLOSE TGT_TABLE_CUR;

-- Drop schema
SET stmtDropSchema = char('DROP SCHEMA ' concat
char(schemaName) concat ' RESTRICT');
EXECUTE IMMEDIATE stmtDropSchema;
END#

This procedures have some limitations, for instance views are not supported.
Nevertheless, it is a good start to all that need to clone a schema or a table.

./M6

Layer Intromission

I'm using JTOpen JDBC driver to access AS/400 via DB2.
It works pretty well, but unfortunately it tries to be more than it should be, a JDBC driver that allows a Java application to access a DB2 database.

It interacts with the AS/400 system in a way it should not. For instance, when it's time to change the password, the driver pops-up a notification window stating that it's time to change the password and asks the user if he wishes to do it now.
If this feature seems nice, let me tell you that it is totally wrong!
First, it is a JDBC driver, not an AS/400 system administration application.
Second, this kind of promiscuity among layers has disastrous results.

When one uses a JDBC driver one expects that it operates with the database and nothing else.
When JTOpen starts getting "smart" by interacting with the system, awkward things happen.
For instance, an entire critical system can fail because the driver hang on a "password will expire soon, do you wish to change it now" message and waits for user action instead of doing what it is suppose to do: database work.
Please not that this is not a "password expired" message but a "you must change your password within x days, do you wish to do it now" question.

Here is a real example of such problem: I executed a command line batch which was hold by the JTOpen driver "would you like to change your password now" question.

JTOpen "Would you like to change your password now" User Dialog Box Question.

I got lucky because the command was issued by me, but what if it was issued by a scheduler on a Saturday morning? It would only be noticed Monday morning and a critical job would not have been performed because the JDBC driver messed up.

./M6

DbVisualizer for DB2 on iSeries

In a previous post I refered that I would be using SQuirreL SQL to access DB2 on AS/400 unless I find some limitiations.
Well, such limitation have been found. I don't know why, but the connection to AS/400 through JTOpen times out every time I run a SQL command, looks like it has a 30 seconds timeout.
I was unable to overcome this, since the driver option for that had no effect when SQuirreL SQL has no option to reconnect automatically nor to keep the session alive.
This has become extremely annoying, uncomfortable, and finally, unusable.

This forced me to search for another solution. And this time it was DbVisualizer. I've used it a couple of times before and I liked it. It's also a Java application and the JTOpen configuration was straight forward.

I'm using it and I'm loving it. The timeout problem simply does not exist. I found DbVisualizer better in many ways, specially in the user interface, witch is better designed.
As before, I'll stick with DbVisualizer, unless I find some limitations or something better comes up.

./M6

SQuirreL SQL for DB2 on iSeries

In the data migration project I'm currently in, the data is located on an AS/400 and I need to access it trough a DB2 connection.
The iSeries Navigator sql interface has some interesting features, like the visual query explain, but it lacks some basic features, like syntax highlight and saving properties changes.
Therefor I've decided to use something else.

I've downloaded SQuirreL SQL and installed it. SQuirreL SQL supports any JDBC connection, since it is written in Java, and during installation I've selected the JDBC for iSeries AS/400 connection.

Then I downloaded JTOpen, the OS/400 and i5/OS JDBC. The installation is quite simple, just unzip the file and update the CLASSPATH to refer the .jar file, I'm using jt400.jar driver.

When SQuirreL SQL starts, it automatically detects the JTOpen(AS/400) driver, so all I had to do was to create a database connection using that driver. I did some configurations on the connection, in particular I only load, and cache, the necessary schemas for my work.

I already had tried SQuirreL SQL with JTOpen to access AS/400 before. It was an older version and it crashed too many times to be really useful.
But this time it seems to be stable, so I'll stick with it for now, unless I find some limitations or something better comes up.

Update: some limitations have came up, and I changed to DbVisualizer.

./M6