Error Initializing JAgent. JAgent will not be initialized and about to return in AIX

Error Initializing JAgent. JAgent will not be initialized and about to return in AIX 

Problem:

2016-01-xx 03:02:59 [main] INFO  JAgentWSMain - About to call initialize on the WebService

2016-01-xx 03:02:59 [main] INFO  JAgentWSMain - Loading agent-spring-ws.xml ###############

2016-01-xx 03:03:00 [main] INFO  AgentInfoImpl - Using IP address 00.00.00.00 to connect to the Manager Web Service

2016-01-xx 03:03:00 [main] INFO  AgentInfoImpl - Reading Manager port 8830 from mgr.prm file.

2016-01-xx 03:03:01 [main] INFO  AgentInfoImpl - OEM Enabled ######

2016-01-xx 03:03:01 [main] INFO  JAgentWSMain - Error Initializing JAgent. JAgent will not be initialized and about to return.

org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'agentBootstrapper': Injection of resource dependencies failed; nested exception is java.lang.NoClassDefFoundError: javax.xml.bind.JAXBException

        at org.springframework.context.annotation.CommonAnnotationBeanPostProcessor.postProcessPropertyValues(CommonAnnotationBeanPostProcessor.java:297)

Solution:

cd OGG_HOME

echo $JAVA_HOME

export JAVA_HOME=/usr/java/java71_64

export PATH=$JAVA_HOME/bin:$PATH

./ggsci



info all

stop extract *

stop replicat *

stop jagent

stop mgr

start mgr

start jagent

start extract *

start replicat *
 
info all



Program     Status      Group       Lag at Chkpt  Time Since Chkpt



MANAGER     RUNNING

JAGENT      RUNNING

 

oracle 13c oem is availbable for download

ORACLE 13c oem is availbable for download:

Oracle Enterprise Manager Cloud Control 13c Release 1 (13.1.0.0) 

http://www.oracle.com/technetwork/oem/enterprise-manager/downloads/index.html

Installing Java8 on AIX

Installing Java8 on AIX

Download Java8 from IBM site

 Java8_64.jre.8.0.0.10.tar.gz

 Java8_64.sdk.8.0.0.10.tar.gz

 Move these folders to /u01/java8/

 gunzip < Java8_64.jre.8.0.0.10.tar.gz | tar xvf -

 gunzip < Java8_64.sdk.8.0.0.10.tar.gz | tar xvf -

login as root user

 run

 smitty installpha

  or

 smitty 

Enter the directory path where software is extract

/u01/java8/

Enter

In the next screen

Select

Accept new license agreements

Press Enter

To make from No To Yes , Press TAB on keyboard

Enter to start the Java8 installation

Check the installation Summary on the screen.

Press F10 if all is success and to exit from the installation screen.

OGG-01091 Unable to open file MGR.rpt in AIX

OGG-01091 Unable to open file MGR.rpt in AIX - Oracle Golden gate Installation discrepancies between major operating systems

By default, manager get started when you select start manager check box during the installation process, however we see this is not the case in AIX operating system

In our case we see manager process did not come up started when installation is done successful

$ ./ggsci



Oracle GoldenGate Command Interpreter for Oracle

Version 12.1.2.1.1 .............

AIX .........., ...........

Operating system character set identified as ..............



Copyright (C) ............... All rights reserved.







GGSCI (hostname) 1> info all



Program     Status      Group       Lag at Chkpt  Time Since Chkpt



MANAGER     STOPPED





GGSCI (hostname) 2> start mgr

Manager started.



GGSCI (hostname) 3>

Source Context :

  SourceModule            : [ggstd.util.file]

  SourceID                : [/../../../../../../gglib/ggstd/fileutl.c]

  SourceFunction          : [ggOpenFile(const char *, const char *)]

  SourceLine              : [772]



201*-**-** 11:01:54  ERROR   OGG-01091  Unable to open file "/../../../../../../../dirrpt/MGR.rpt" (error 2, No such file or directory).



201*-**-** 11:01:54  ERROR   OGG-01668  PROCESS ABENDING.



GGSCI (hostname) 3>



GGSCI (hostname) 4> create subdirs



Creating subdirectories under current directory /../../../../../../



Parameter files                /../../../../../../: already exists

Report files                   /../../../../../../: created

Checkpoint files               /../../../../../../: created

Process status files           /../../../../../../: created

SQL script files               /../../../../../../: created

Database definitions files     /../../../../../../: created

Extract data files             /../../../../../../: created

Temporary files                /../../../../../../: created

Credential store files         /../../../../../../: created

Masterkey wallet files         /../../../../../../: created

Dump files                     /../../../../../../: created





GGSCI (hostname) 6> start mgr

Manager started.

all is well thereafter...

Error in invoking target 'ntcontab.o' of makefile in OBIEE 11.1.1.9.0 Installation

Error in invoking target 'ntcontab.o' of makefile  in OBIEE 11.1.1.9.0 Installation

Error:

Exception Name: MakefileException

Exception String: Error in invoking target 'ntcontab.o' of makefile '/../../../../../network/lib/ins_net_client.mk'.

Solution:

df -g /tmp/



df -h /tmp/

check the directories that are taking huge disk space  using

cd /tmp/

du -ag  $PWD |sort -n -r|head

check if we have atleast 2GB free space, I made 4 GB space available.. then installation was successfull..

How to check if database is exadata

select count(*) from (select distinct cell_name from gv$cell_state);

returns a number > 0, then the database runs on Exadata storage.


Non exadata:
-----------------

SQL> select count(*) from (select distinct cell_name from gv$cell_state);



  COUNT(*)

----------

         0



SQL> select distinct cell_name from gv$cell_state;



no rows selected

RA-31633: unable to create master table '' ORA-30511: invalid DDL operation in system triggers

PROBLEM:
-----------

expdp schemas=USERNAME directory=DUMP_DIR content=metadata_only dumpfile=USERNAME.dmp logfile=USERNAME.log userid=USERNAME/USERNAME



Export: Release 12.1.0.1.0 - Production on ..........



Copyright (c) 1982, 2013, Oracle and/or its affiliates.  All rights reserved.



Connected to: Oracle Database 12c ......................

With the Partitioning, OLAP, Advanced Analytics and Real Application Testing options

ORA-31626: job does not exist

ORA-31633: unable to create master table "USERNAME.SYS_EXPORT_SCHEMA_XX"

ORA-06512: at "SYS.DBMS_SYS_ERROR", line 95

ORA-06512: at "SYS.KUPV$FT", line 1031

ORA-30511: invalid DDL operation in system triggers

ORA-06512: at line 1288

ORA-30511: invalid DDL operation in system triggers

Solution:
--------------

SQL@USERNAME>Purge DBA_RECYCLEBIN;



DBA Recyclebin purged.



SQL@USERNAME>purge recyclebin;



Recyclebin purged.

If does not works Restart database and do the above steps again... All should be fine....

ORA-00060: deadlock detected while waiting for resource

Problem:
-------------
ORA-00060: deadlock detected while waiting for resource


Solution:
----------
Go to database trace folder /../../../trace/

cat alert_DBNAME.log |grep -i deadlock

ORA-00060: Deadlock detected. See Note at My Oracle Support for Troubleshooting ORA-60 Errors. More info in file /../../../../trace/SID_ora_XXX.trc

open one of the trace file and see what causing issues

see the "[Transaction Deadlock]" section to see if it is really an ORACLE errors

see the "Deadlock graph: " for any deadlocks

check for the Information , DML statements for the current (blocker) session and OTHER waiting sessions..

DBA do not have to do anything unless they are oracle errors, most of the cases i see , these are application issues. Design application and/or tune DMLs are the solutions.

ORA-00039: error during periodic action ORA-04036: PGA memory used by the instance exceeds PGA_AGGREGATE_LIMIT

Caused By: java.sql.SQLException: ORA-00039: error during periodic action ORA-04036: PGA memory used by the instance exceeds PGA_AGGREGATE_LIMIT

We see these errors frequently in oracle 12c databases,  PGA_AGGREGATE_LIMIT can be set manually or by default and its a limiting value on the aggregate PGA memory that instance can use max.
the sessions (running untunable DMLs) using the most memory will be terminated and the above said errors get reported.

An incident is a single occurrence of a problem, an incident is created for each problem occurrence

The sessions that experience ORA-04036:  errors will get terminated and these sessions current sql statements can be seen in trace files.


What is the fastest solution to this?

We either run one of the below alter to solve

alter system set pga_aggregate_limit=0 scope=both;

or

alter system set pga_aggregate_limit=10G scope=both;

Does it solves the problem permanently?  I dont think it will.. so what should we do to solve this problem . there are two ways either DML producers (Developers) tune their queries, if they say something is wrong with db  .. then as  DBA has to dig deep down to find the DML statements that causing issues...

here it is one of the method i use to find the queries that cause these issues..

We have a good tool known as ADRCI

login into adrci

adrci> show incidents

Find incidents that are causing 04036 issues

show incident -mode detail -p "incident_id=XXXX"

it shows the incident files with KEY INCIDENT_FILE, these are actually trace files..

Get the trace file names and open trace files to see causes.... the trace files shows more than required info.. session id, time, everything .. more importantly it shows  like

----- Current SQL Statement for this session (sql_id=xxxxxxxxxxxxxxxxx) -----
 we get not just SQL_ID but whole SQL statements that caused these issues... what else we want,.. let  developers know DML statements     and or  get these statements tuned..

 Hope it helps u all..

 Open for suggestions and comments.

How to create service name in oracle database or Add service name to single instance database

Service name can be different from database name or instance name (also know as SID). Default service name is created with same as database name. Service name is like an alias that allow us to connect the db, it can be recorded in tnsnames.ora file or may not be recorded in tnsnames.ora file, it depends what connection method we use to connect.
we can create more than one service name for single database. The way we create service name in single instance db is different from RAC. the method below shown is for single instance db.

Let me explain why did we required to do the below...  Most people may wonder why we do have more than 8 character size service name or instance name. Oracle may not recommends but does not stop from creating instance_name with more than 8 character length string. We have a db with more than 8 character length and users want to connect db with 8 character length string with no db downtime, hence we did the below.. and it works......

In real production systems, we use service names as it allows to manage resources, workloads more effectively.
*real strings are changed*

We can also use dbms_service package .

connect sys as sysdba



show parameter service_name



NAME                                 TYPE        VALUE

------------------------------------ ----------- ------------------------------

service_names                        string      MYPRODUCTIONDB



show parameter db_name



NAME                                 TYPE        VALUE

------------------------------------ ----------- ------------------------------

db_name                              string      MYPRODUC





SQL> alter system set service_names='MYPRODUCTIONDB,MYPRODUC' scope=both;



System altered.



SQL> alter system register;



System altered.



SQL> sho parameter service_name



NAME                                 TYPE        VALUE

------------------------------------ ----------- ------------------------------

service_names                        string      MYPRODUCTIONDB,MYPRODUC





SQL> !tnsping MYPRODUCTIONDB

It works as it has entry in tnsnames.ora file

SQL> !tnsping MYPRODUC

TNS-03505: Failed to resolve name

--- It did not work as we have no entry for this in our tnsnames.ora file.

We still conntect db with new service name

SQL> CONNECT system/passwordscret@myhost.com:1521/MYPRODUC

Connected.

SQL> CONNECT system/passwordscret@//myhost.com/MYPRODUC

Connected.

SQL> CONNECT system/passwordscret@//myhost.com/MYPRODUC

Connected.

SQL>

The method we used to connect db is known as EZCONNECT or JDBC connection . EZCONNECT is very easy connect naming method and does not require any edits in tnsnames.ora file.

ORA-12518: TNS:listener could not hand off client connection

ORA-12518: TNS:listener could not hand off client connection



Listener stop, start .. it resolved the isseus.. if nothing helps ..try adding the below to listner. ora file.
DIRECT_HANDOFF_TTC_LISTENER=OFF

Listener stop, start .

connect to the db

alter system register;

sqlplus user/pwd@db

Ensure you have sufficent value is set for processes init parameter, change it to max lets say 1000 or 2000

Always ensure db is configured with right AMM  or properly configured SGA





Its been awhile posting a blog entry, I am back to blogging.. hopefully you all like my blogs and post your comments as always.