Tracking web services calls in WebCenter logs

When invoking web services from WebCenter applications, it may be useful to track all the requests and responses generated in the application. In order to achieve this, some configuration in WebLogic console must be done:

1. Log in to WebLogic console as administrator user.

2. Navigate to WebCenter Portal managed server.

3. Select Configuration -> Server Start tab.

4. In the Arguments box, paste the following commands:


  -Dcom.sun.xml.ws.transport.http.client.HttpTransportPipe.dump=true  -Dweblogic.security.SSL.verbose=true -Dcom.sun.net.ssl.checkRevocation=false



5. Save and restart the managed server.

After that, in the out log file there will be traces with all the requests to web services, as well as their responses.

For instance:

---[HTTP request - http://<WS_Host>:6073/Service/CustomerManagement/Endpoint]---
Content-type: text/xml; charset=utf-8
Soapaction: "/Service/CustomerManagement.serviceagent/CustomerManagementEndpoint/GetServiceBillingAccounts"
Accept: text/xml, multipart/related, text/html, image/gif, image/jpeg, *; q=.2, */*; q=.2
<?xml version='1.0' encoding='UTF-8'?>

[..]

---[HTTP response - http://<WS_Host>:6073/Service/CustomerManagement/Endpoint - 200]---
null: HTTP/1.1 200 OK
Content-type: text/xml; charset="utf-8"
Content-length: 32026
Connection: keep-alive
Server: Jakarta-HttpComponents-NIO/1.1
Date: Thu, 17 Sep 2015 17:44:42 GMT
<?xml version="1.0" encoding="UTF-8"?>

It is important to note that this configuration is not recommended in Production environments, as it negatively affects the performance of the WebCenter platform.

Acknowledgement: Thanks to Roberto Fuertes Peña for his help.

Data source drivers for ADF applications

Recently I found an error in an ADF 12c application in Production which was prompting error popups to users all the time. In the managed server log files, the following trace was registered:

oracle.jbo.SQLStmtException: JBO-29114 ADFContext is not setup to process messages for this exception.
[..]
Caused By: java.sql.SQLException: Statement cancelled, probably by transaction timing out

Firstly, I checked data source connectivity from WebLogic console, but the result was successful.

Then I found this post and this one, where the source of the problem was attributed to the data source driver. Despite the common practice of assigning JDBC XA drivers to data sources connections, this should not be used with Oracle Fusion applications, as it is documented here, because they are not compatible, "XA data sources close all cursors upon commit" and that produces errors. As that data source was using that type of thin XA driver (oracle.jdbc.xa.client.OracleXADataSource), I decided to change it.

As the list of driver types is only displayed when creating a new one, nd I did not know the exact name, I decide to create a new mock data source, just to get that value, which was:

Description: Oracle's Driver (Thin) for Instance connections; Versions:Any
Name: oracle.jdbc.OracleDriver

Once I knew the name, I edited the current data source with that driver name. However, I could not activate changes, because an error was received, it was related to not been able to update de data source configuration file.

The workaround consists in changing manually that configuration file. Navigate to the following folder in the domain server:

<adf_domain_dir>/config/jdbc 

Create a backup copy of the file with the data source name and edit the original one. Just update the parameter related to driver-name:

<driver-name>oracle.jdbc.OracleDriver</driver-name>

After this, the error disappeared from the production environment.

References

Don't Use Oracle's Driver (Thin XA) to Define WebLogic Data Source for ADF Applications
ADF: Resolving random JBO-27122 and closed statement errors
What You May Need to Know About JDBC Data Source for Oracle WebLogic Server

WebCenter Managed Servers taking too long to start up

Issue 

 

In an Oracle WebCenter Portal installation, the process of starting a managed server took around 40 minutes. Checking the start logs, there was a long time between the login and the following line.

 [EL Info]: 2013-11-06 19:35:25.773--ServerSession(645650875)--file:/SOFT/software/middleware/product/11.1.1/oracle_common/modules/oracle.jps_11.1.1/jps-internal.jar_JpsDBDataManagerV2_nonJtaDataSource=594822561 login successful

<Nov 6, 2013 7:58:58 PM CET> <Warning> <J2EE> <BEA-160140> <Unresolved optional package references (in META-INF/MANIFEST.MF): [Extension-Name:
oracle.apps.common.resource

 Also, in the managed server log, it was registered a big delay in database connections:

The thread is waiting for data from DataBase (as the admin as managed server show the same thread)
[ACTIVE] ExecuteThread: '0' for queue: 'weblogic.kernel.Default (self-tuning)'" id=23 idx=0x60 tid=24740 prio=5 alive, in native, daemon

 In order to find out the source of this problem, an AWR (Automatic Workload Repository) was requested to DBA. The report showed there was a single query that was taking 400 seconds to execute during server startup:

SELECT LOGID, DN, ACTION, APPLICATIONDN, ATTRNAME, CREATEDATE, ATTRVALUE, VERSION FROM JPS_CHANGELOG WHERE DN LIKE :1 ESCAPE :2 ORDER BY LOGID DESC

That query is against the JPS_CHANGELOG table in the OPSS (Oracle Platform Security Services) schema, where WebCenter security is located. It is a log where all the updates in WebCenter permissions are registered. The size of the table was checked, getting a result of 600.000 rows. Therefore, it needed to be purged.

Solution

 

Oracle explained how to maintain that table here. Basically, the following query should be executed in the OPSS schema:

SQL>delete from jps_changelog where createdate < (select(max(createdate) - 1) from jps_changelog);

SQL>Commit;

It is also recommended to execute the DMS_STATS package to gather database storage statistics.

After that purge:

1. Start of the servers, including AdminServer, only took 4 minutes.
2. JPS_CHANGELOG table size was 2.000 rows.
3. AWR report showed that the query was taking only 40 seconds.

In case you are not sure if WebLogic security is maintained in databse, perform the following steps:

1. Login to Enterprise Manager as administrator.
2. Select WebLogic domain.
3. Navigate to Security -> Security Provider Configuration.
4. In the screen the database storage would appear.

Conclusion 

 

If WebCenter Portal has security delegated in an OPSS schema, it is important to program a periodic purge of the JPS_CHANGELOG table. Otherwise, platform performance would decrease progressively, showing an increase in the startup time of the servers.

References

Oracle® Fusion Middleware Application Security Guide 11g Release 1 (11.1.1):

http://docs.oracle.com/cd/E21764_01/core.1111/e10043/cfgauthr.htm#CHDEGJJA

Oracle Support Service Request:

SR 3-8069279071 : My custom portal node spends more 25 min to startup