Apache http Failed To Start In R12.1.3 after clone

Hiya Buddies!!!

We would like to share this below error which is very normal during cloning. Here, we are facing this issue because of some OS upgrade/incompatibilities from source to target.

We got the below error while starting the EBS Apps services. While start the application services, we found that apache was getting failed as shown below error log details:

Executing service control script:

/u01/test/inst/test_tech/admin/scripts/adapcctl.sh start

Timeout specified in context file: 100 second(s)

script returned:

****************************************************

You are running adapcctl.sh version 120.7.12010000.2

Starting OPMN managed Oracle HTTP Server (OHS) instance ...

adapcctl.sh: exiting with status 204

adapcctl.sh: check the logfile /u01/test/inst/test_tech/logs/appl/admin/log/adapcctl.txt for more information ...

 ==

 opmn id=tech:6208

    0 of 1 processes started.

ias-instance id=test_tech.tech.non-tech.com

++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++

--------------------------------------------------------------------------------

ias-component/process-type/process-set:

    HTTP_Server/HTTP_Server/HTTP_Server/

Error

--> Process (index=1,uid=778177156,pid=9722)

 failed to start a managed process after the maximum retry limit

Log:

 /u01/test/inst/test_tech/logs/ora/10.1.3/opmn/HTTP_Server~1.log

07/15/18-21:08:04 :: adapcctl.sh: exiting with status 204

=============================================================

    /u01/test/inst/test_tech/logs/ora/10.1.3/opmn/HTTP_Server~1.log

18/07/15 21:08:02 Start process

--------

Can't stat executable /u01/test/inst/test_tech/ora/10.1.3/Apache/Apache/bin/.apachectl

18/07/14 03:05:56 Start process

--------

/u01/test/inst/test_tech/ora/10.1.3/Apache/Apache/bin/apachectl startssl: execing httpd

Status:

Web Entry Point Services   Oracle HTTP Server test_tech          adapcctl.sh      Failed

Solution:

For this we need to apply a patch in 10.1.3 Oracle Home as shown below:

Please find the below steps:

1. We need to source oracle_home to 10.1.3

[tech:] $cd $ORA_CONFIG_HOME/10.1.3/

$ls -ltr *.env

or we can export ORACLE_HOME to 10.1.3 manually...

2. We need to attached oracle home, this step can be ignored also...

$cd $ORACLE_HOME/oui/bin

$./runInstaller -silent -attachHome -invPtrLoc $INST_TOP/admin/oraInst.loc ORACLE_HOME=/u01/test/apps/tech_st/10.1.3  -ignoreSysPrereqs ORACLE_HOME_NAME=test_10_1_3_home

$$ORACLE_HOME/OPatch/opatch lsinventory -invPtrLoc \  $INST_TOP/admin/oraInst.loc

3. We need to apply this patch 12722501 on 10.1.3 Oracle Home (OPatch).

$cd /u01/patch/OPatch/12722501

#$ORACLE_HOME/OPatch/opatch apply -invPtrLoc $INST_TOP/admin/oraInst.loc

$ORACLE_HOME/OPatch/opatch apply -silent -invPtrLoc \ $INST_TOP/admin/oraInst.loc -ocmrf \ /appdata/delphix/postclone/apps/backup/ocm_response.rsp

Here, -ocmrf is oracle configuration manager response file, it is included with release of opatch.

4. $ORACLE_HOME/OPatch/opatch lsinventory -invPtrLoc \ $INST_TOP/admin/oraInst.loc

5. Run the Apps env file.

6. Run autoconfig on Apps Tier.

$cd $ADMIN_SCRIPTS_HOME

./adautocfg.sh appspass=$APPS_PWD

7. Start the application services.

Note: This solution is not for all cases if apache is not starting. This is only as per MOS.

Some more useful links:

How to Study and Prepare for Exams

How to Enable or Disable Archive log mode in oracledatabase 11g

How to export query output to Excel in Oracle SQLDeveloper

How to check notepad++ backup files

Regards,

TechOrNontech Team

OID , OAM and ASM log files location and details:

Hiya Buddies!!!

We would like share some real time log file path which is very important for any organizations or DBAs.

Actually, we all think that how can we remember the every log files path. It is true but while working for long time on same team or company , you can keep remember for almost path of the alert log files.

And also, the fact is everyone forget the stuff. But the thing is somebody keep revision or keep working on same things, so they can remember the stuff easily.

Otherwise, there is no short-cut to remember everything... hahahh

We will keep update this post as getting some log files which are not easily found.

This is only sample, path may depends on server to server or projects.

Note: The way to find out either log file or any files by name is either you can use "find" command or "locate" command. Locate command works only if it is installed/configured on your server.

For using find command better to go to base directory and try, or the directory where software is installed. You can find out by using "grep process" command as ps -ef|grep ASM.

$find . -name "filename" -exec ls -ltr {} \;

If you are not sure about the exact file name then you can use "*" as shown below:

$find . -name "alert*" -exec ls -ltr {} \;

$locate <file_name>

OID log:

In below path we are checking for first ods1 log. If there are many or two servers then we can go to two directory back and get the list of servers under Server directory.

[tech&non-tech:] $ pwd

/u01/oracle/oid/domain/oid_domain/servers/wls_ods1/logs

[tech&non-tech:] $

[tech&non-tech:] $ pwd

/u01/ admin/ oid_domain/ servers  ### We have changed starting path

[tech&non-tech:] $

OAM logs:

[tech&non-tech:] $ pwd

/u01/oracle/oam/domain/oam_domain/servers/oam_server1/logs

[tech&non-tech:] $

ASM alert log file:

$ pwd

/u01/app/oracle/diag/asm/+asm/+ASM1/trace

oracle@tech&non-tech:/u01/app/oracle/diag/asm/+asm/+ASM1/trace$

$ tail -100f alert_+ASM1.log

Some more useful links:

How to Unlock EBS apps user in R12

RMAN Issues And Solutions Troubleshooting

Database Prechecks Script before clone

Delphix error: DBREFRESH failed due to an errorduring execution: Failed to create mount

Thanks,

How to Unlock EBS apps user in R12

Hiya Buddies!!!

We would like to share a simple and easy topic as how to unlock EBS apps user in R12.1.3

There are two methods, we can do it either from front-end or back-end.

Here, we would like to share the code which we have done in Production and worked fine. So, there is no doubt.

But we need to change the user and password as per our request username and password.

Here, for example in our case username is ora-data which status is as LOCKED.

If you want to change it from back-end then you can run the below command from apps user.

SQL> select name from v$database;

NAME

---------

ORA-DATA

SQL> show user

USER is "APPS"

SQL>

SQL>

DECLARE

    status BOOLEAN;

BEGIN

    status := FND_USER_PKG.CHANGEPASSWORD(username => 'ora-data', newpassword => 'welcome1234');

    IF status

    THEN

                   DBMS_OUTPUT.PUT_LINE('Password changed and user unlocked');

    commit;

    ELSE

        DBMS_OUTPUT.PUT_LINE('Error while setting password');

    END IF;

END;

/

PL/SQL procedure successfully completed.

SQL>

This type of request is usually used to come from the either application team or developer team. So, this is a type of daily DBA task activities.
And this is very easy way to change because as we know that changing from front-end , it is like headache because front-end always takes time to load and sometimes java issue doesn't support etc...

Hope this may useful and helpful. For any concerns please let us know either in comment box or Contact Us above.

Some more useful links:

Performance Issue and Solution

How to start MRP process on Dataguard -- stand by database

RMAN Issues and Solutions - (ORA-00020) and (ORA-00245)

Unable to find an Output Post Processor service to post-process request
Regards,
TechOrNontech Team

How to run concurrent program gather schema statistics front-end in R12

Hiya Buddies!!!

It is very easy steps to submit the concurrent request for Gather Schema Statistics from front-end.

So, we would like explain here as How to run concurrent program Gather Schema Statistics.

1. We need to login EBS (Oracle Applications) front-end url as SYSADMIN user and Responsibility as System Administrator

If you are not sure about EBS url, you can login to back-end host and follow below

Steps:

$hostname;id

ebs.oracle.com;applmgr

$. ./APPSEBS_ebs.env

$echo $TWO_TASK

EBSPROD

$grep login $CONTEXT_FILE

<you will get the ebs url as output...>

2. Here, we need to submit a Request, follow navigation as shown below:

 Navigate to: Request > Run.

In Name: Select as : Gather Schema Statistics

i.e. Press "f11" then type "Gather Scehma Statistics" thne Press "CTRL and F11" together.

You will get a pop-up with some options such as Schema Name, Estimate_percent, Degree etc...

3. Enter the appropriate parameters. This can be run for specific schema's by specifying the schema name or entering 'ALL' to gather statistics for every schema in the database.

4. Submit the gather schema statistics program.

Estimate_percent

Percentage of rows to estimate. If you left empty it will take by default to 10%.

The valid range is 0-99. A higher percentage will be more accurate, but it will take longer time to run.

Degree

Here, we enter the Degree of parallelism. If we do not enter, it will take by default values to like min(cpu_count, parallel_max_servers).

And modifying the degree of parallelism on a table can cause the plan to change.

In a easy language, increasing the degree of parallelism is likely to make full table scans appear cheaper and more attractive

while reducing it will make Full Table Scans look less attractive.

Example which we have done...

Degree - 4

% (percentage) - 40

Some more useful links:

Performance Issue and Solution

How to start MRP process on Dataguard -- stand by database

RMAN Issues and Solutions - (ORA-00020) and (ORA-00245)

Unable to find an Output Post Processor service to post-process request

Thanks...

TechorNontech Team

Oracle APPS DBA Real Time Issues and Solutions

Hiya Buddies,

I would like to share some real time issue and solution which we have faced during the EBS integration recently.

These are very common issues also which may helpful here to all...

1. Issue –

Suppose, our Concurrent Manager processing are running slow then what we have done and what we need to check.

As per our investigation we found as 2000 request was submitted one at time. So, CM cannot cache as request are coming one at time

What we did and got Fix:

1) Reduced Conflict Resolution Manager Sleep Time to 10

2) Increase Cache Size to 20 for standard manager

2) Issue -

This issue we get almost time and it is very common : java.lang.OutOfMemoryError: Java heap space & Intermittent HTTP-500 Errors

Solution:

We got the fix issue by Increase Java Heap & number of oacore JVM’s

a)      <oacore_jvm_start_options oa_var="s_oacore_jvm_start_options">-server -

verbose:gc -Xmx1024M -Xms512M -XX:MaxPermSize=256M -XX:NewRatio=2  -XX:+PrintGCT

imeStamps -XX:+UseTLAB -XX:+UseParallelGC  -XX:ParallelGCThreads=2  -Dcom.sun.ma

nagement.jmxremote -Djava.security.policy=$ORACLE_HOME/j2ee/oacore/config/java2.

policy -Djava.awt.headless=true -Dhttp.webdir.enable=false -Doracle.security.jaz

n.config=/u01/apps/inst/apps/ORA-DATA/ora/10.1.3/j2ee/oacore/config/j

azn.xml</oacore_jvm_start_options>

b)  <oacore_nprocs oa_var="s_oacore_nprocs">2</oacore_nprocs>

3) Issue -

Log file details:

java.io.IOException: Broken pipe

13/01/19 11:12:04.849 html: Servlet error

java.io.IOException: Broken pipe

        at sun.nio.ch.FileDispatcher.write0(Native Method)

        at sun.nio.ch.SocketDispatcher.write(SocketDispatcher.java:29)

        at sun.nio.ch.IOUtil.writeFromNativeBuffer(IOUtil.java:104)

        at sun.nio.ch.IOUtil.write(IOUtil.java:75)

        at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:334)

        at java.nio.channels.Channels.write(Channels.java:60)

        at java.nio.channels.Channels.access$000(Channels.java:47)

        at java.nio.channels.Channels$1.write(Channels.java:134)

        at com.evermind.server.http.AJPOutputStream.endRequest(AJPOutputStream.java:117)

        at com.evermind.server.http.AJPRequestHandler.run(AJPRequestHandler.java:317)

        at com.evermind.server.http.AJPRequestHandler.run(AJPRequestHandler.java:199)

        at oracle.oc4j.network.ServerSocketReadHandler$SafeRunnable.run(ServerSocketReadHandler.java:260)

For this above issue we found work-around and Fix as:

– Apache pid should be on Local File System than

s_lock_pid_dir - /u01/apps/inst/apps/ORA-DATA/pids/10.1.3/Apache/logs

s_pids_dir - /u01/apps/inst/apps/ORA-DATA/pids

s_web_pid_file - /u01/apps/inst/apps/ORA-DATA/pids/10.1.3/Apache/httpd.pid

4)Issue –

Suppose, our Workflow Notification Mailer is running either slow or not working then what we need to do.

Work-around:

a)  Workflow Manager Have System Hold Fix Manager Before Resetting Counters

Solution:

Wfmgsms And Wfmgsmd Managers Have System Hold Fix Manager Before Resetting Counters [ID 1378284.1] -- ref.

SQL> update FND_CONCURRENT_QUEUES

set ENABLED_FLAG='N'

where CONCURRENT_QUEUE_NAME like 'WF%GSM%'

SQL> commit;

b)  Notification Mailer slow/not running

oracle.apps.fnd.wf.mailer.NotificationFormatter$FormatterSAXException:

  Problem obtaining the HTML content -> oracle.apps.fnd.wf.common.HTTPClientException:

  Unable to invoke method HTTPClient.HTTPResponse.listHeaders caused by:

  java.net.SocketTimeoutException: Read timed out

Solution:

Profile “WF: Workflow Mailer Framework Web Agent” set.

select t.PROFILE_OPTION_ID ID, z.USER_PROFILE_OPTION_NAME Profile, nvl(v.PROFILE_OPTION_VALUE,'Replace with non-load balanced URL') Value, decode(v.level_id, 10001,'Site',

                            10002,'Application',

                            10003,'Responsibility',

                            10004,'User',

                            10005,'Server',

                            10006,'Organization') "LEVEL"

from fnd_profile_options t, fnd_profile_option_values v, fnd_profile_options_tl z

where (v.PROFILE_OPTION_ID (+) = t.PROFILE_OPTION_ID)

and (z.PROFILE_OPTION_NAME = t.PROFILE_OPTION_NAME)

and (t.PROFILE_OPTION_NAME in

('APPS_FRAMEWORK_AGENT','WF_MAIL_WEB_AGENT'));

To set the WF_MAIL_WEB_AGENT profile :

- Stop Workflow Mailer and Agent Listener service

- Change the "WF: Workflow Mailer Framework Web Agent" [WF_MAIL_WEB_AGENT] profile option to point to a physical Web Server host address that is non-load balanced.

- Rebuild mailer queue with:

SQL>@$FND_TOP/patch/115/sql/wfntfqup.sql APPS <APPS schema passwd> APPLSYS•Start Workflow Mailer and Agent Listener service.

-Test scenario with a new notification.

select * from wf_notifications

check mail_status=’SENT’ for notification

5)Issue -

FRM-41839 Disk I/O error on temporary record buffer file /tmp/AAxxe893.tmp

Solution:

We need to Increase the /tmp size to 8 GB

Some more useful link:

RMAN Issues and Solutions - (ORA-00020) and(ORA-00245)

Difference between fnd_conc_clone.setup_clean and cmclean

How To Start and Stop Informatica Services in Linux/Unix

Iprocurement Home Page Error : You have encounteredan unexpected error. Please contact System Administrator of assistance

Thanks,

How to run SQLT for a sql_id in Oracle

Hiya Buddies, Topic: How to run SQLT for a particular sql_id

We would like to share a very important topic which may useful for both either core DBA or apps DBA.

Usually we used to get the request from users to provide SQLT from a sql_id.

It is very important and easy also but we need to be very careful.

Sometimes it may throw error also.  Same way I also got the

Request from users and ran the SQLT script but got error also which I would like to share with you all here…

How to run SQLT for a particular sql_id. Suppose, our sql_id is ora-data1234.

# pwd

/home/oracle/sqlt/sqlt/run

oracle@host:/home/oracle/sqlt/sqlt/run

# ls -ltr sqltxtract.sql

-rw-r—r--   1 oracle   dba         4431 Oct 30  2014 sqltxtract.sql

oracle@host:/home/oracle/sqlt/sqlt/run

#sqlplus ‘/as sysdba'

SQL>@sqltxtract.sql ora-data1234

It will ask for password SQLTXTRACT which may same as username or different if you have made any changes… In our case it is same as username.

If you provide wrong password,  no need to worry.  It will ask again username and password.

Now, also you may get the below error for tablespace, better to keep monitoring log file.

Issue:

ORA-01691: unable to extend lob segment SQLTXP

Solution:

SQL>Alter tablespace USERS add datafile ‘/path/TEST/datafile/path/datafile/users.9326.978199625’ size 1024 M;

 File ID  File Name                                         Megabytes

-------- -------------------------------------------------- ---------

     293 /path/TEST/datafile/path/datafile/dat       500

         afile/users.9326.978199624

     275 /path/TEST/datafile/path/datafile/dat       500

         afile/users.9326.978199623

Below is the few lines which you will get once it completed.

SQLDX files have been created.

Archive:  sqlt_s53282_sqldx.zip

  Length      Date    Time    Name

---------  ---------- -----   ----

   670545  11-12-2018 07:49   sqlt_s53282_sqldx_ora-data1234_csv.zip

   521831  11-12-2018 07:50   sqlt_s53282_sqldx_2017923558923286652_force_csv.zip

   662100  11-12-2018 07:51   sqlt_s53282_sqldx_table_csv.zip

    42639  11-12-2018 07:51   sqlt_s53282_sqldx_global_csv.zip

    10141  11-12-2018 07:51   sqlt_s53282_sqldx_ora-data1234_log.zip

---------                     -------

  1907256                     5 files

  adding: sqlt_s53282_sqldx.zip (stored 0%)

SQLTXTRACT completed.

SQL>

Note: If you face tablespace issues, just need to add/resize datafile. It will take care automatically, no need to restart the SQLT command.

Some more useful links:

How to deploy RPD in OBIEE 11g

Unable to find an Output Post Processor service to post-processrequest

Concurrent Managers are down error insert_fcp failed due to ORA-01403 no data found, ORA-06512

oracle apps dba interview questions and answers for experiencedasked Top MNC company