I just updated oem12c 12.1.0.2.0 -> 12.1.0.3.0 and updated few agents.
So – few new problem also spotted…

$emctl status agent
Oracle Enterprise Manager Cloud Control 12c Release 3
Copyright (c) 1996, 2013 Oracle Corporation. All rights reserved.
---------------------------------------------------------------
Agent Version : 12.1.0.3.0
OMS Version : (unknown)
(...)
Collection Status : Collections enabled
Heartbeat Status : Agent is blocked
Blocked Reason : Plugin mismatch found between agent and repository. Please perform an agent resync to unblock the agent. Mismatched plugins: [ The plug-in is oracle.sysman.db
(...)
---------------------------------------------------------------
Agent is Running and Ready

 

Let’s repair it:

agent is blocked

001. Agent is blocked

001. One of the agent after update of the OMS come to “blocked” status (under blackout).
We can see them see in the databases / hosts or simply from the agents list.
Setup -> Manage Cloud Control -> Agents


 

002. Let’s check this agent on the target machine:

SERVER_NAME agent12c:/home/oracle $emctl status agent
Oracle Enterprise Manager Cloud Control 12c Release 3
Copyright (c) 1996, 2013 Oracle Corporation. All rights reserved.
---------------------------------------------------------------
Agent Version : 12.1.0.3.0
OMS Version : (unknown)
Protocol Version : 12.1.0.1.0
Agent Home : /u01/app/oracle/product/12.1.0.2/agent_inst
Agent Binaries : /u01/app/oracle/product/12.1.0.2/core/12.1.0.2.0
Agent Process ID : 2835
Parent Process ID : 1587
Agent URL : https://SERVER_NAME:3873/emd/main/
Repository URL : https://oem12c:4900/empbs/upload
Started at : 2014-04-22 13:17:59
Started by user : oracle
Last Reload : (none)
Last successful upload : (none)
Last attempted upload : (none)
Total Megabytes of XML files uploaded so far : 0
Number of XML files pending upload : 1
Size of XML files pending upload(MB) : 0
Available disk space on upload filesystem : 33.37%
Collection Status : Collections enabled
Heartbeat Status : Agent is blocked
Blocked Reason : Plugin mismatch found between agent and repository. Please perform an agent resync to unblock the agent. Mismatched plugins: [ The plug-in is oracle.sysman.db
Last attempted heartbeat to OMS : 2014-04-22 13:36:04
Last successful heartbeat to OMS : (none)
Next scheduled heartbeat to OMS : 2014-04-22 13:39:04
---------------------------------------------------------------
Agent is Running and Ready

Plugin mismatch? Resynchronization is needed – so come back to oem12c panel.


 

002. Resynchronize agent

Enter into agent screen (click on his name – displayed in previous picture).
From menu select Agent -> Resynchronization

 

002a – Agent Resynchronization confirmation

 

002b – Operation scheduled

Agent resync
A job has been submitted for resync. Please monitor progress of the resync job from jobs page: …….

 

002b – Resynchronization log

It”s a look over resync job – this will take a small amount of time…

 


 

003. When resynchronization will end – try agent status now?

SERVER_NAME agent12c:/home/oracle $emctl status agent
Oracle Enterprise Manager Cloud Control 12c Release 3
Copyright (c) 1996, 2013 Oracle Corporation. All rights reserved.
---------------------------------------------------------------
Agent Version : 12.1.0.3.0
OMS Version : 12.1.0.3.0
Protocol Version : 12.1.0.1.0
Agent Home : /u01/app/oracle/product/12.1.0.2/agent_inst
Agent Binaries : /u01/app/oracle/product/12.1.0.2/core/12.1.0.2.0
Agent Process ID : 2835
Parent Process ID : 1587
Agent URL : https://SERVER_NAME:3873/emd/main/
Repository URL : https://oem12c:4900/empbs/upload
Started at : 2014-04-22 13:17:59
Started by user : oracle
Last Reload : (none)
Last successful upload : (none)
Last attempted upload : (none)
Total Megabytes of XML files uploaded so far : 0
Number of XML files pending upload : 31
Size of XML files pending upload(MB) : 0.07
Available disk space on upload filesystem : 33.36%
Collection Status : Collections enabled
Heartbeat Status : Agent is blocked
Blocked Reason : Plugin information does not match with repository, agent resync required.
Last attempted heartbeat to OMS : 2014-04-22 14:06:06
Last successful heartbeat to OMS : (none)
Next scheduled heartbeat to OMS : 2014-04-22 14:09:07

---------------------------------------------------------------
Agent is Running and Ready

Plugin mismatch is dissapear, but resync is needed again?
Use oem12c console and do it again. (Agent -> Resynchronization)

 


 

004. Resynchronization are completed. What next? Let’s check agent page:

004 – Agent under blackout

So resynchronization are complete, we need to unblock agent to connect them into oms.
Check agent on this host:

SERVER_NAME agent12c:/home/oracle $emctl status agent
Oracle Enterprise Manager Cloud Control 12c Release 3
Copyright (c) 1996, 2013 Oracle Corporation. All rights reserved.
---------------------------------------------------------------
Agent Version : 12.1.0.2.0
OMS Version : 12.1.0.3.0
Protocol Version : 12.1.0.1.0
Agent Home : /u01/app/oracle/product/12.1.0.2/agent_inst
Agent Binaries : /u01/app/oracle/product/12.1.0.2/core/12.1.0.2.0
Agent Process ID : 13539
Parent Process ID : 13357
Agent URL : https://SERVER_NAME:3873/emd/main/
Repository URL : https://oem12c:4900/empbs/upload
Started at : 2014-04-22 14:44:25
Started by user : oracle
Last Reload : (none)
Last successful upload : 2014-04-22 15:14:52
Last attempted upload : 2014-04-22 15:14:52
Total Megabytes of XML files uploaded so far : 0.13
Number of XML files pending upload : 0
Size of XML files pending upload(MB) : 0
Available disk space on upload filesystem : 33.36%
Collection Status : Collections enabled
Heartbeat Status : Ok
Last attempted heartbeat to OMS : 2014-04-22 16:19:47
Last successful heartbeat to OMS : 2014-04-22 16:19:47
Next scheduled heartbeat to OMS : 2014-04-22 16:20:47

---------------------------------------------------------------
Agent is Running and Ready

Looks fine, but still blocked in OMS.

 


 

005 – unblock agent

005. Unblock agent: (pic is not from this process – i miss it and created a replacement from different machine)
Select Agent -> Contol -> End Blackout

005a – unblock agent

Mark blocked agent and click on “Stop”.

 


 

006 – Agent is up, running and contacted with OMS!

After a while – be patient! This is not immediatelly! Agent will be marked as up and running.
This can take few minutes before operation will be completed.