ASM does not mount disk group after server reboot.

Today when checking a recently installed server with Oracle Linux 7.9 and ASM 19.17 with ASMLib, we found that when performing any reboot on the server, ASM could not mount the disks.

OS and kernel version:

[grid@srv01 ~]$ uname -a
Linux srv01 5.4.17-2136.316.7.el7uek.x86_64 #2 SMP Mon Jan 23 18:37:26 PST 2023 x86_64 x86_64 x86_64 GNU/Linux

[grid@srv01 ~]$ cat /etc/*-release | grep PRETTY
PRETTY_NAME="Oracle Linux Server 7.9"

ASM version:

[grid@srv01 ~]$ asmcmd -V
asmcmd version 19.17.0.0.0

Disks used in multipath ASM for DATA and RECO:

[root@srv01 ~]# multipath -ll
RECO01 (3600a0980384b5545464d42) dm-4 NETAPP  ,LUN C-Mode
size=100G features='4 queue_if_no_path pg_init_retries 50 retain_attached_hw_handle' hwhandler='1 alua' wp=rw
|-+- policy='service-time 0' prio=50 status=active
| `- 33:0:0:0 sdd                8:48   active ready running
`-+- policy='service-time 0' prio=10 status=enabled
  `- 34:0:0:0 sdc                8:32   active ready running
DATA01 (3600a0455244b5545464d43) dm-5 NETAPP  ,LUN C-Mode
size=100G features='4 queue_if_no_path pg_init_retries 50 retain_attached_hw_handle' hwhandler='1 alua' wp=rw
|-+- policy='service-time 0' prio=50 status=active
| `- 33:0:0:1 sdf                8:80   active ready running
`-+- policy='service-time 0' prio=10 status=enabled
  `- 34:0:0:1 sde                8:64   active ready running

When performing a disk rescan, ASM mounted the DATA and RECO disk groups without errors, when performing a server reboot the problem occurred again:

[root@srv01 ~]# /etc/init.d/oracleasm scandisks
[root@srv01 ~]# /usr/sbin/oracleasm listdisks

When analyzing the logs, we verified that the oracleasm.service service started before the iscsid.service concluded, in this way the disks became unavailable so that the ASM could perform the MOUNT operation of the disks groups:

/var/log/messages

Fix applied: Modification of oracleasm.service file, changing the startup order of services to ensure that CRS ohasd.service is started after oracleasm.service has successfully discovered all ASM disks:

[root@srv01 ~]# cat /lib/systemd/system/oracleasm.service
[Unit]
Description=Load oracleasm Modules
Requires=multipathd.service iscsi.service multi-user.target  <========
After=multipathd.service iscsi.service multi-user.target    <=========
[...]

[root@srv01 ~]# systemctl daemon-reload

Reboot the server to effectively test the fix:

[root@srv01 ~]# init 6

After correction, disks are mounted in ASM without error after server reboot:

[grid@srv01 ~]$ asmcmd
ASMCMD> lsdg
State    Type    Rebal  Sector  Logical_Sector  Block       AU  Total_MB  Free_MB  Req_mir_free_MB  Usable_file_MB  Offline_disks  Voting_files  Name
MOUNTED  EXTERN  N         512             512   4096  4194304    102400   102296                0          102296              0             N  DATA/
MOUNTED  EXTERN  N         512             512   4096  4194304    102412   102316                0          102316              0             N  RECO/
ASMCMD> exit

Help tech note:

Oracle Linux 7: ASMLib-Managed Disks on Multipathed iSCSI Targets are not Discovered 
after Server Reboot (Doc ID 2214152.1)