The Setup

We have an Oracle server with database links created to multiple database servers. The target servers also being Oracle. There is a stored procedure written which queries the data present in certain tables in these linked servers and essentially copies them to the tables in the local database. Some of these insert-select statements take few minutes to complete, depending on the data present in the source tables.

One important point to remember is that the select queries are not grouped by the link server which they are querying from. For example, let's consider we have two database links L1 and L2, the body of our stored procedure resembled this...

INSERT INTO LOCAL_TABLE_1
SELECT * FROM L1.SOURCE_TABLE_1;

INSERT INTO LOCAL_TABLE_2
SELECT * FROM L2.SOURCE_TABLE_2;

INSERT INTO LOCAL_TABLE_3
SELECT * FROM L1.SOURCE_TABLE_3;

The Issue

Intermittently, the procedure started failing by throwing an exception at the third insert-select statement stating ORA-03150 error.

The cause

A bit of searching around on Google hinted that this could be a connectivity issue between the Oracle server and the linked server. Upon further digging in, there is a 15 minute connection idle timeout on the target server. The target server was closing the connection after 15 minutes of inactivity. So whenever query 2, where we were querying L2 database link, is taking more than the threshold, the connection is closed to L1.

We had the liberty to group the queries together based on the database links so we modified the stored procedure slightly like the following and didn't face the problem anymore after this.

INSERT INTO LOCAL_TABLE_1
SELECT * FROM L1.SOURCE_TABLE_1;

INSERT INTO LOCAL_TABLE_3
SELECT * FROM L1.SOURCE_TABLE_3;

INSERT INTO LOCAL_TABLE_2
SELECT * FROM L2.SOURCE_TABLE_2;

In conclusion, Oracle is creating a new connection with L1 and when query related to L2 takes up lot of time, the linked server is closing the connection from its end as it detected inactivity. When the stored procedure finished executing the second query and reaches the third one, Oracle is attempting to use the same connection created earlier at the beginning of the stored procedure execution. This is resulting in an error as the connection is already closed from the other end.