解决oracle

解决oracle oem启动问题 由重庆ORACLE培训思庄科技提供Troubleshoot Database Control Startup Issues

In this Document

PurposeLast Review DateInstructions for the ReaderTroubleshooting DetailsWhat is Database Control?InstallationDirectory structureHow to use?Log files that aid in troubleshootingSteps to troubleshoot Database Control startup issuesKnown issuesReferences

Applies to:

Enterprise Manager Grid Control - Version: 10.1.0.2 to 10.2.0.1Information in this document applies to any platform.

Purpose

Provide troubleshooting steps for Database Control startup issues.

Last Review Date

January 15, 2008

Instructions for the Reader

A Troubleshooting Guide is provided to assist in debugging a specific issue. When possible, diagnostic tools are included in the documentto assist in troubleshooting.

Troubleshooting Details

What is Database Control?

The Web-based Database Control serves as the primary tool for managing an Oracle database. From the Oracle Enterprise Manager, onecan perform administrative tasks such as creating schema objects (tablespaces, tables, and indexes), managing user security, backing upand recovering the database, and importing and exporting data. One can also view performance and status information about themanaged database.

Installation

Database Control is by default installed if “Use Database Control for Database Management” option is used on the “Select DatabaseManagement Option” page when installing the Oracle Database Server. If this option is not initially selected the Database Control can becreated afterwards using emca tool. Steps to create/drop/recreate can be found in Note 278100.1Additional information regarding emca can be found at:Oracle® Enterprise Manager Advanced Configuration10g Release 3 (10.2.0.3.0)Part Number B40002-02

http://download-uk.oracle.com/docs/cd/B16240_01/doc/em.102/b40002/structure.htm#CACBBFGI

Directory structure

-> Files that are common and shared among all instances of the database are stored in the following directory of the Oracle Database 10ghome:

ORACLE_HOME/sysman

-> Files that are unique to each instance of the database are stored in following directory of the Oracle Database 10g home:ORACLE_HOME/hostname_sid/sysman/ (for a single instance database)ORACLE_HOME/nodename_sid/sysman/ (for a cluster database)

If a hostname_sid directory does not exist in the Oracle Database 10g home directory, then Oracle Enterprise Manager 10g DatabaseControl was never configured for the database instance.-> Log files for diagnosing issues are located under:ORACLE_HOME/hostname_sid/sysman/log/ORACLE_HOME/nodename_sid/sysman/log/

How to use?

At the end of a successful database creation, Database Control is automatically launched in the web browser and the database is started.To use the Database Control any other time the service needs to be running.Check if the Database Control service is running by executing:ORACLE_HOME/bin/emctl status dbconsolethe output should be:

Oracle Enterprise Manager 10g Database Control Release <5 digit version>Copyright (c) 1996, 2005 Oracle Corporation. All rights reserved.http://:5500/em/console/aboutApplicationOracle Enterprise Manager 10g is running.

------------------------------------------------------------------Logs are generated in directory /d01/oracle/product/10.2.0/db_1/hostname_sid/sysman/logIf the Database Control service is not running start it with:ORACLE_HOME/bin/emctl start dbconsoleTo stop the Database Control service run:ORACLE_HOME/bin/emctl stop dbconsole

A complete list of available emctl commands can be obtained by just typing:ORACLE_HOME/bin/emctl

Log files that aid in troubleshooting

- ORACLE_HOME/hostname_sid/sysman/log/emdb.nohup - contains the standard output from the Console main Java process at eachstart/exit (dbconsole and agent). It is appended with each new output of the Console main Java process.- ORACLE_HOME/hostname_sid/sysman/log/emagent.log and emagent.trc - main agent log & trace files

- ORACLE_HOME/hostname_sid/sysman/log/emoms.log and emoms.trc - main management service log & trace files- ORACLE_HOME/hostname_SID/sysman/log/emdctl.log and emdctl.trc - emctl log & trace files

- ORACLE_HOME/hostname_SID/sysman/log/emagentfetchlet.trc and emagentfetchlet.log - log and trace file for Java FetchletsAdditional information related to log files can be found under Note 403928.1, Note 229624.1.

Steps to troubleshoot Database Control startup issues

Common startup issues:1. Environment variables;2. SYSMAN/DBSNMP issues3. Timezone

4. Network

5. Configuration (wrong port assignment, wrong connection string)

Troubleshooting steps:

1. Check if environment variables are set correctly. Mainly, check, ORACLE_HOME, PATH, LD_LIBRARY_PATH, LANG.2. Check if the SYSMAN/DBSNMP account is open. To check, connect to database as SYS and run:SQL> select username, account_status from dba_users where username in ('SYSMAN','DBSNMP');and the output:

USERNAME ACCOUNT_STATUS-------------- ------------------------DBSNMP OPENSYSMAN OPENemagent.trc errors2008-01-19 11:20:21,231 [HttpRequestHandler-28730188] ERROR conn.ConnectionService verifyRepositoryEx.433 - Invalid ConnectionPool. ERROR = ORA-28000: the account is locked

3. Check the timezone set in the environment. If the timezone does not match one of the values in

ORACLE_HOME/sysman/admin/supportedtzs.lst the dbcontrol agent will not start. When checking dbcontrol status returns: \"EM Deamon isnot running\".emdb.nohup errors----- Wed Jul 25 22:31:53 2007::property 'agentTZregion' in

'/usr/pkg/oracle/product/10.2.0/db//sysman/config/emd.properties' contains an invalid value of 'TZ set to '.Agent start up can not proceed.This value might have been manually modified to be an incorrect value.This value needs to be set toone of the values listed in '/usr/pkg/oracle/product/10.2.0/db/sysman/admin/supportedtzs.lst'. Execute 'emctl config agent getTZ' and see ifthis is an appropriate value. -----4. Check the OS network configuration:

static IP (no DHCP generated IP);

hostname must not contain \"_\" (underscore character);nslookup, ping must resolve the fully qualified name;\"hosts\" file entries pattern:

lookup and reverse lookup must work;IPv6 is not supported;

5. Check the database network configuration:

check \"lsnrctl status\" if shows same listener details as ORACLE_HOME/network/admin/listener.ora

check TNS status with tnsping utility

emoms.trc errorsORA-2005-07-04 12:23:08,120 [XMLLoader0] ERROR conn.ConnectionService verifyRepositoryEx.418 - Invalid Connection Pool. ERROR= Listener refused the connection with the following error: ORA-12514, TNS:listener does not currently know of service requested inconnect descriptor The Connection descriptor used by the client was: (DESCRIPTION=(ADDRESS_LIST=(ADDRESS=(PROTOCOL=TCP)(HOST=)(PORT=)))(CONNECT_DATA=(SERVIC_NAME=)))

6. Check the configuration files are correct.

ORACLE_HOME//sysman/emd.properties. Check properties:

REPOSITORY_URL=http://:/em/upload/EMD_URL=http://:/emd/main

ORACLE_HOME//sysman/emoms.properties. Check properties:oracle.sysman.eml.mntr.emdRepConnectDescriptor - must have a valid connection stringoracle.sysman.eml.mntr.emdRepPort=oracle.sysman.eml.mntr.emdRepDBName=oracle.sysman.emSDK.svlt.ConsoleServerPort=oracle.sysman.emSDK.svlt.ConsoleServerHost=oracle.sysman.emSDK.svlt.ConsoleServerHTTPSPort=If none of the above helps check the Known Issues list below.

Known issues

Generic platform

1. EM Deamon is not running. Database Control starts successfully, however checking the status shows \"EM Deamon is not running\". Alsochecking the agent status shows that agent is not running.//sysman/log/emdctl.trc shows:

2005-11-06 18:04:40 Thread-3840 ERROR main: nmectl.c: nmectl_validateTZRegion, agentTZoffset=120, and testTZoffset for GMT:0 donot matchSolution:a) Set the desired time zone at the OS level:Windows: Control Panel->Date&Time->Time ZoneLinux/Unix: export TZ=

the timezone select must correspond to one of the timzones in ORACLE_HOME/sysman/admin/supportedtzs.lstb) Stop the dbconsole

ORACLE_HOME/bin/emctl stop dbconsolec) Run:

ORACLE_HOME\\bin\\emctl config agent getTZ

This may return a diffrent timezone than the one set in step 1.ORACLE_HOME\\bin\\emctl config agent updateTZ

This will update the ORACLE_HOME\\\\sysman\\config\\emd.properties file with the correct timezone.d) Start dbconsole

ORACLE_HOME/bin/emctl start dbconsole

2. GIM-00104: Health check failed to connect to instance.

ORACLE_HOME//sysman/log/emagent.trc shows:

2006-05-04 13:17:29 Thread-2206875655 ERROR fetchlets.healthCheck: GIM-00104:Health check failed to connect to instance.GIM-00090: OS-dependent operation:open failed with status: 24GIM-00091: OS failure message: Too many open filesGIM-00092: OS failure occurred at: sskgmsmr_7

2006-05-04 13:17:29 Thread-2206875655 ERROR engine: [oracle_database,tmprod_tmprod2,health_check] : nmeegd_GetMetricDatafailed : Instance HealthCheck initialization failed due to one of the following causes: the owner of the EM agent process is not same as theowner of the Oracle instance processes; the owner of the EM agent process is not part of the dba group; or the database version is not

10g (10.1.0.2) and above.

2006-05-04 13:17:30 Thread-22062039 ERROR http: snmehl_connect: failed to create socket: Too many open files (error = 24)Solution:Check Note 368612.13. Error starting ORMI-Server. Unable to bind socket: Address already in use. Trying to start db control fails without an obvious reason.ORACLE_HOME//sysman/log/emdb.nohup shows:

----- Mon Nov 6 10:34:13 2006::Console Launched with PID 3441 at time Mon Nov 6 10:34:13 200606/11/06 10:34:16 Error starting ORMI-Server. Unable to bind socket: Address already in useSolution:Check Note 398499.1, Note 419586.1, Note 438504.1, Note 3561.14. Unable to determine local host from URL.emctl start dbconsole shows:

EMD_URL=http://:/emd/mainSolution:Check A HREF=\"/metalink/plsql/showdoc?db=NOT&id=266027.1&blackframe=1\" >Note 266027.1, Note 343748.15. OC4J Configuration Issue: $ORACLE_HOME/oc4j/j2ee/OC4J_DBConsole_host_sid not foundemctl start dbconsole fails with the following error:

OC4J Configuration Issue: $ORACLE_HOME/oc4j/j2ee/OC4J_DBConsole_host_sid not foundThere are three cases for this issue:

a) ORACLE_HOME variable needed to run emctl is set to the wrong database home. Set the right value and retry the process.

b) Network changes. If the hostname where Database Control was created in the first place is not resolvable anymore startup will fail withabove error.Example:

Database configuration folders:

ORACLE_HOME/oc4j/j2ee/OC4J_DBConsole_myhost_sidORACLE_HOME/myhost_sid

network changes made hostname \"myhost\" to be unresolvable. Instead of \"myhost\" the hostname resolves to \"myhost.mydomain.com\"emctl will not find:

ORACLE_HOME/oc4j/j2ee/OC4J_DBConsole_myhost.mydomain.com_sidORACLE_HOME/myhost.mydomain.com_sid

To resolve the issue database control needs to be recreated using the correct hostname.Check Note 278100.1 for steps to create db control.

c) Database Control was not configured. Check Note 278100.1 for steps to create db control.6. Perl errors. Getting the following errors when starting the dbcontrol:perl: warning: Setting locale failed.

perl: warning: Please check that your locale settings:LC_ALL = (unset),LC__FASTMSG = \"true\

LANG = \"En_US\"

are supported and installed on your system.

perl: warning: Falling back to the standard locale (\"C\")SolutionThis is a label view environment problem. In the local environment do the following:Unset the LANG variable.Stop and restart the database.

7. Start Dbconsole Shows Errors With Wrong Ps Option on Unix AIX, HP, SOLARIS> emctl start dbconsole

Oracle Enterprise Manager 10g Database Control Release 10.2.0.1.0Copyright (c) 1996, 2005 Oracle Corporation. All rights reserved.http://:1158/em/console/aboutApplicationps: unknown output format: -o cmdps: illegal option -- -ps: unknown output format: -o ls

usage: ps [ -aAdeflcjLPy ] [ -o format ] [ -t termlist ][ -u userlist ] [ -U userlist ] [ -G grouplist ][ -p proclist ] [ -g pgrplist ] [ -s sidlist ]'format' is one or more of:

user ruser group rgroup uid ruid gid rgid pid ppid pgid sid taskid pri opri pcpu pmem vsz rss osz nice class time etime stime f s c lwp nlwppsr tty addr wchan fname comm args projid project pset

Starting Oracle Enterprise Manager 10g Database Control ..................... started.SolutionCheck Note 358479.18. Starting dbcontrol fails. emdb.nohup shows:

----- ::Console Launched with PID 12031 at time -----Exception in thread \"main\" java.util.zip.ZipException: No such file or directoryat java.util.zip.ZipFile.open(Native Method)at java.util.zip.ZipFile.(ZipFile.java:112)at java.util.jar.JarFile.(JarFile.java:127)SolutionCheck Note 312652.19. Startup dbcontrol is making too many \"__JDBC__\" entries in LISTENER log:

26-SEP-2005 12:09:00 * (CONNECT_DATA=(CID=(PROGRAM=)(HOST=__jdbc__)(USER=))(SERVICE_NAME=beta.query))* (ADDRESS=(PROTO =tcp)(HOST=hostname)(PORT=52163)) * establish * beta.query * 0

26-SEP-2005 12:09:00 * (CONNECT_DATA=(CID=(PROGRAM=)(HOST=__jdbc__)(USER=))(SERVICE_NAME=beta.query))* (ADDRESS=(PROTO =tcp)(HOST=hostname)(PORT=521)) * establish * beta.query * 0SolutionCheck Note 336177.110. 'emctl start dbconsole' takes ages to start. Emagent process is actually running, although one is unable to access dbcontrol from thebrowser. emagent.trc shows:

2007-09-16 10:48:16 Thread-1290 WARN vpxoci: OCI Error -- ErrorCode(6550): ORA-06550: line 1, column 65: PLS-00201: identifier

'DBMS_AQADM' must be declared ORA-06550: line 1, column 65:PL/SQL: Statement ignored

SQL = \" BEGN :succ_sub := 0; dbms_aqadm.creat\"...

LOGIN = dbsnmp/@(DESCRIPTION=(ADDRESS=(PROTOCOL=TCP)(HOST=)(PORT=1521))(CONNECT_DATA=(SID=FSSYS)))SolutionCheck Note 458834.111. dbconsole having hostname starting with letter \"u\" fails. This is because of DBCONSOLE DOES NOT WORK HAVINGA HOSTNAME STARTING WITH \"U\". Bug is fixed in 10.2.0.2 patchset.Apply patchset 10.2.0.2 or if not possible:

a) Save the file $ORACLE_HOME\\host_SID\\sysman\\config\\emd.properties to emd.properties.origb) Update the file $ORACLE_HOME\\host_SID\\sysman\\config\\emd.properties, replacing \\ with / in thefollowing line:For example change:

omsRecvDir=d:\\oracle\\product\\10.2.0\\db_1\1_db0\\sysman\\recvto

omsRecvDir=d:/oracle/product/10.2.0/db_1/ukp001_db0l/sysman/recvc) Bounce DB Control

12. SEVERE: Cannot start Database Control. The following ports are already in use: [EM agent port:3938]a) Check which ports are available for use for this dbcontrolb) Run the creation of the dbcontrol with using this port numbers:

ORACLE_HOME\\bin\\emca -config all db -repos recreate -AGENT_PORT -DBCONTROL_HTTP_PORT -RMI_PORT -JMS_PORT The default port numbers are:AGENT_PORT: 3938

DBCONTROL_HTTP_PORT: 5500 or 1158RMI_PORT: 5520JMS_PORT: 50

13. The dbconsole cannot be initialized correctly and the logfile $ORACLE_HOME/_/sysman/log/emoms.trc showsthe following error:

ORA-12516, TNS:listener could not find available handler with matching protocol stackThe Connection descriptor used by the client was:

< oracle.sysman.eml.mntr.emdRepConnectDescriptor defined in

$ORACLE_HOME/_/sysman/config/emoms.properties>SolutionCheck Note 458308.114. Generic Time zone issues.

Check Note 338556.1, Note 304585.1, Note 332123.1, Note 461918.1.Platform specificAIX5L

1. dbcontrol start fails with:./emctl start dbconsole

bin/emctl[336]: unlimited: 0403-009 The specified number is not valid for this command.This happens because AIX 5L O/S itself as documented in AIX Version 4.3 to 5L Migration Guide:http://www.redbooks.ibm.com/redbooks/pdfs/sg246924.pdf on Page 195.

When the emctl script try to see the value of ulimit. If it's set to unlimited, it throws this exception and continue starting the DBConsole.Solutiona) You can simply ignore the warning

b) Set the value of the ulimit to a value either than unlimited.2. unable to star dbcontrol. Main error in emdb.nohup:

+12139 [ Unable to alloc heap of requested size, perhaps the maxdata value is too small - seeREADME.HTML for more information. ]+12140 [ **Out of memory, aborting** ]+12141 [ ]+12142 [ ]

+12142 [ *** panic: JVMST017: Cannot allocate memory in initializeMarkAndAllocBits(markbits1) ]+12143 /u01/app/oracle/CPSS/10.2.0/jdk/bin/java[3]: 1253762 IOT/Abort trap(coredump)

+12144 ----- Thu Jul 19 18:27:17 2007::DBConsole exited at Thu Jul 19 18:27:17 2007 with returnvalue 134. -----+12145 ----- Thu Jul 19 18:27:17 2007::DBConsole has exited due to an internal error -----+12146 ----- Thu Jul 19 18:27:17 2007:: - checking for corefile at/u01/app/oracle/CPSS/10.2.0/abc.xyz.com_sid/sysman/emd -----+12147 ----- Thu Jul 19 18:27:17 2007::Restarting DBConsole. -----+12148 ----- Thu Jul 19 18:27:17 2007::Console Launched with PID 1015900 at time Thu Jul 1918:27:17 2007 -----+12149 [ Unable to alloc heap of requested size, perhaps the maxdata value is too small - seeREADME.HTML for more information. ]+12150 [ **Out of memory, aborting** ]SolutionIncrease ulimit resources for the user starting the dbcontrol.3. dbcontrol fails to start. emagent.trc shows:

2006-06-14 14:06:01 Thread-1872 ERROR engine: [oracle_database,,health_check] : nmeegd_GetMetricData failed :2006-06-14 14:06:11 Thread-1562 ERROR pingManager: nmepm_pingReposURL: Error in request response. code = 400. text = 2006-06-14 14:06:16 Thread-1634 ERROR fetchlets: Could not load library '/u01/app/oracle/product/10.2.0/db/lib32/libnmcfhc.so' for reason 'rtld:0712-001 Symbol main was referenced from module /u01/app/oracle/product/10.2.0/db/lib32/libnmcfhc.so(), but a runtime definition of thesymbol was not found.

rtld: 0712-001 Symbol nmeusb_StringBuffer_new was referenced from module /u01/app/oracle/product/10.2.0/db/lib32/libnmcfhc.so(), buta runtime definition of the symbol was not found.SolutionCheck Note 378104.14. The Refresh time on the database control home page shows two hours behind the standard time for

Europe/Copenhagen on AIX platform. How to change the time to correct value? (this w/a can be implemented for other timezones regionsas long as the corresponding AIX timezone is known).Check Note 860955.1Windows Server 2003

1. Starting the dbcontrol fails with:

The OracleDBConsoleCIMISYU service terminated with service-specific error 1 (0x1)An error occured while trying to initialize the service.Solutiona) Apply the latest Patch 6012744 - 10.2.0.3.0 Patch6 for Microsoft Windows (x).

b) All bugs included in Patch 5846378 are also included in Patch 6012744, since these are cumulative patches.2. dbcontrol fails to start. emagent.trc shows:

2005-08-26 11:53:56 Thread-4 ERROR pingManager: nmepm_pingReposURL: Cannot connect tohttp://:5501/em/upload/: retStatus=-1

2005-08-26 11:53:57 Thread-4 WARN http: snmehl_connect: connect failed to (:5501): No connection could be madebecause the target machine actively refused it.SolutionThis is an installation issue. During the installation there are some files missing.The files

oc4j\\j2ee\\oc4j_applications\\applications\\em\\em\\WEB-INF\\lib\jaroc4j\\j2ee\\oc4j_applications\\applications\\em\\em\\WEB-INF\\lib\\ohw.jaroc4j\\j2ee\\oc4j_applications\\applications\\em\\em\\WEB-INF\\lib\\share.jar

are missing. In the most cases you needs to create the directory WEB-INF\\lib manually.

As a workaround, copy the above 3 files to '...\\WEB-INF\\lib' and restarted dbconsole. The files are located in $ORACLE_HOME\\jlib.

References

Note 266027.1 - Problem: Startup: Emctl Start Dbconsole Fails with Agent port missing in EMD_URLNote 278100.1 - How To Drop, Create And Recreate DB Control In A 10g Database

Note 343748.1 - Problem: Startup: Error starting Database Control, dbconsole - Unable to determine local host from URLNote 3561.1 - Problem: Startup OMS: Oms Startup Fails With Integration Class Not Found

Note 368612.1 - Problem: Startup: DB Control Agent Crashes: Gim-00091 OS failure Message: Too Many Open FilesNote 398499.1 - Problem: Startup: EM Database Control Has Stopped Working and Unable to Start AgainNote 403928.1 - How to cycle the DB Control emdb.nohup file in $ORACLE_HOME/host_sid/sysman/log

Note 419586.1 - Problem: Startup: Cannot Start dbconsole and log Shows 'ORMI-Server address is already being used'Note 438504.1 - EMCA or DB Control (DBConsole) Fails with Error starting ORMI-ServerNote 452284.1 - How to manage DB Console Log and Trace files

http://download-uk.oracle.com/docs/cd/B16240_01/doc/em.102/b40002/structure.htm#CACBBFGI