logo
Agent are displayed in agent list as unreachable.
Try to upload data?

[oraagent@hostname ~]$ emctl upload agent
Oracle Enterprise Manager Cloud Control 12c Release 3
Copyright (c) 1996, 2013 Oracle Corporation. All rights reserved.
---------------------------------------------------------------
EMD upload error:The agent is overloaded [current requests: 128]

Situation described in this post are spotted on 2014-06-02 but i can’t time to check and backup logs, so i back to analyse on 2014-06-03, but logs been overwritten and i don’t know what is reason of this situation…


001. Agent are displayed in agent list as unreachable.
Let’s check them?

001. Agent is unreachable

[oraagent@hostname ~]$ emctl status agent
Oracle Enterprise Manager Cloud Control 12c Release 3
Copyright (c) 1996, 2013 Oracle Corporation. All rights reserved.
---------------------------------------------------------------
Agent Version : 12.1.0.3.0
OMS Version : 12.1.0.3.0
Protocol Version : 12.1.0.1.0
Agent Home : /u01/app/oraagent/agent12c/agent_inst
Agent Binaries : /u01/app/oraagent/agent12c/core/12.1.0.3.0
Agent Process ID : 4916
Parent Process ID : 4856
Agent URL : https://hostname:3873/emd/main/
Repository URL : https://oem12c:4900/empbs/upload
Started at : 2014-05-25 00:32:40
Started by user : oraagent
Last Reload : (none)
Last successful upload : 2014-05-25 00:36:24
Last attempted upload : 2014-05-25 00:36:24
Total Megabytes of XML files uploaded so far : 0.09
Number of XML files pending upload : 0
Size of XML files pending upload(MB) : 0
Available disk space on upload filesystem : 59.87%
Collection Status : Collections enabled
Heartbeat Status : Ok
Last attempted heartbeat to OMS : 2014-05-25 00:36:26
Last successful heartbeat to OMS : 2014-05-25 00:36:26
Next scheduled heartbeat to OMS : 2014-05-25 00:37:26

---------------------------------------------------------------
Agent is Running and Ready

Agent is running and ready, but his health status are strange.
Is few days later and i see, that agent… (on 2014-06-03):
– is started on 2014-05-25 00:32:40 (this is an situation when auto-tuning restarted it, i think…)
– after few minutes later he upload data to repository
– he set next upload date (one minute later after last upload)
– ne never uploaded newest data?
– this look, that entire agent process are broken – this is still 0 XML files pending upload, after few days?
– logs are gone, i can’t see now what was a reason of this…


002. Let’s try to upload?

[oraagent@hostname ~]$ emctl upload agent
Oracle Enterprise Manager Cloud Control 12c Release 3
Copyright (c) 1996, 2013 Oracle Corporation. All rights reserved.
---------------------------------------------------------------
EMD upload error:The agent is overloaded [current requests: 128]

003. Why this agent are overloaded? System have high load? Top…

Tasks: 270 total, 2 running, 268 sleeping, 0 stopped, 0 zombie
Cpu(s): 1.4%us, 0.7%sy, 0.0%ni, 96.3%id, 1.3%wa, 0.1%hi, 0.1%si, 0.0%st
Mem: 16436064k total, 11562908k used, 4873156k free, 553208k buffers
Swap: 17301496k total, 1432828k used, 15868668k free, 7924616k cached

 PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
 4008 oracle 17 0 10.0g 24m 22m S 3.9 0.2 0:00.02 oracle
 4004 root 15 0 11008 1068 680 R 2.0 0.0 0:00.01 top
 6085 root 15 0 352m 35m 15m S 2.0 0.2 44:34.45 orarootagent.bi
 7291 grid 16 0 84520 13m 9976 S 2.0 0.1 8:17.67 tnslsnr
 7631 oracle 15 0 10.0g 66m 62m S 2.0 0.4 10:39.24 oracle
 1 root 15 0 10368 684 572 S 0.0 0.0 0:55.48 init
 2 root RT -5 0 0 0 S 0.0 0.0 0:00.00 migration/0
 3 root 34 19 0 0 0 S 0.0 0.0 0:00.09 ksoftirqd/0
 4 root RT -5 0 0 0 S 0.0 0.0 0:00.00 migration/1
 5 root 34 19 0 0 0 S 0.0 0.0 0:00.16 ksoftirqd/1
 6 root RT -5 0 0 0 S 0.0 0.0 0:00.00 migration/2
 7 root 34 19 0 0 0 S 0.0 0.0 0:01.25 ksoftirqd/2
 8 root RT -5 0 0 0 S 0.0 0.0 0:00.00 migration/3

Looks normal, What is a reason of this…

 


 

004. Repair? Restart solved problem, but… my bad, that i not look in logs but just restarted. I still not known a reason…

[oraagent@hostname ~]$ emctl stop agent
Oracle Enterprise Manager Cloud Control 12c Release 3
Copyright (c) 1996, 2013 Oracle Corporation. All rights reserved.
Stopping agent ... stopped.

[oraagent@hostname ~]$ emctl start agent
Oracle Enterprise Manager Cloud Control 12c Release 3
Copyright (c) 1996, 2013 Oracle Corporation. All rights reserved.
Starting agent ................................. started.

[oraagent@hostname ~]$ emctl status agent
Oracle Enterprise Manager Cloud Control 12c Release 3
Copyright (c) 1996, 2013 Oracle Corporation. All rights reserved.
---------------------------------------------------------------
Agent Version : 12.1.0.3.0
OMS Version : 12.1.0.3.0
Protocol Version : 12.1.0.1.0
Agent Home : /u01/app/oraagent/agent12c/agent_inst
Agent Binaries : /u01/app/oraagent/agent12c/core/12.1.0.3.0
Agent Process ID : 4622
Parent Process ID : 4538
Agent URL : https://hostname:3873/emd/main/
Repository URL : https://oem12c:4900/empbs/upload
Started at : 2014-06-02 15:02:32
Started by user : oraagent
Last Reload : (none)
Last successful upload : 2014-06-02 15:03:02
Last attempted upload : 2014-06-02 15:03:02
Total Megabytes of XML files uploaded so far : 0.08
Number of XML files pending upload : 0
Size of XML files pending upload(MB) : 0
Available disk space on upload filesystem : 59.71%
Collection Status : Collections enabled
Heartbeat Status : Ok
Last attempted heartbeat to OMS : 2014-06-02 15:02:53
Last successful heartbeat to OMS : 2014-06-02 15:02:53
Next scheduled heartbeat to OMS : 2014-06-02 15:03:53

---------------------------------------------------------------
Agent is Running and Ready

 


 

005. Agent is restarted. But is he working now?
Let’s try to upload?

 [oraagent@hostname ~]$ emctl upload agent
Oracle Enterprise Manager Cloud Control 12c Release 3
Copyright (c) 1996, 2013 Oracle Corporation. All rights reserved.
---------------------------------------------------------------
EMD upload completed successfully

005. Agent is uploading data


 

006. Conclusion. What i do wrong?
I not checked logs but just reloaded agent.
They are overwritten. I still don’t know source of this behaviour.

 

.