[RAC][INSTALL] 2_OS설치 및 다양한 환경 설정

 

작업 내용

※ os 설치
※ grid/racdb1,2 유저 생성
※ 기동중인 service 중지
※ hosts 네트워크 설정
※ yum 설정 및 package 설치
※ 공유 스토리지 생성 및 설정
※ grid/racdb1,2 디렉토리 생성 및 .bash_profile 설정
※ 커널 파라미터 설정
※ 유저 리소스 제한 설정

 

설치 파일 

• Oracle Linux ISO 설치 파일 : https://edelivery.oracle.com/

Oracle Software Delivery Cloud

 

OS 설치

 

Oracle Delivery를 통해 다운받은 부팅 디스크 지정

 

 

 

언어 설정

 

옵션설정(1) Localization

 

 

- DATE & TIME 설정

  ▶ 실제 사용자가 사용하는 NTP(Network Time Protocol)에 맞게 설정 

 

- LanguageSupport 설정 (필수 x)

 

옵션설정(2) SOFTWARE

 

- Installation Source

  ▶ 부팅디스크 외에 다른 파일이 있다면 사용 

       (설명 : https://docs.centos.org/en-US/centos/install-guide/SourceSpoke-x86/)

 

- Software selection

Gui 환경에서 설정 필요

 

옵션설정(3) SYSTEM

 

자동설정 외 직접 파티셔닝 설정

 

파티션 별 용량 할당

( swap : 10GiB / /boot 1024 MiB / / : 남은 용량 설정)

 

- before -

 

- After - 

 

Network 및 Hostname 설정

 

- hostname 변경

 

- RAC1 enp0s3 ethernet 설정 (NAT IP) 로 설정 

  ※ 설치시 네트워크 설정 잘못하여 구축 후 아래와 같이 재설정

  ※ RAC2 동일하게 설정하되 10.0.2.11 로 진행

- RAC1 enp0s8 ethernet 설정 (Host-Only IP) 로 설정

  ※ 설치시 네트워크 설정 잘못하여 구축 후 아래와 같이 재설정

  ※ RAC2 동일하게 설정하되 192.168.56.102 로 진행

root 패스워드 설정

 

reboot 후 License 및 최종 확인

 

설치 완료 화면

---

유저 생성 (RAC1,RAC2)

1. 유저 생성

- rac1번 유저생성

[root@ol79rac1 ~]# groupadd dba

[root@ol79rac1 ~]# useradd -g dba grid

[root@ol79rac1 ~]# passwd grid
Changing password for user grid.
New password: grid
BAD PASSWORD: The password is shorter than 8 characters
Retype new password: grid
passwd: all authentication tokens updated successfully.

[root@ol79rac1 ~]# useradd -g dba oracle
[root@ol79rac1 ~]# passwd oracle
Changing password for user oracle.
New password:oracle
BAD PASSWORD: The password is shorter than 7 characters
Retype new password:oracle
passwd: all authentication tokens updated successfully.

------------------------------------------------------------------------
- rac2번 유저생성

[root@ol79rac2 ~]# groupadd dba

[root@ol79rac2 ~]# useradd -g dba grid
[root@ol79rac2 ~]# passwd grid
Changing password for user grid.
New password:grid
BAD PASSWORD: The password is shorter than 8 characters
Retype new password:grid
passwd: all authentication tokens updated successfully.

[root@ol79rac2 ~]# useradd -g dba oracle
[root@ol79rac2 ~]# passwd oracle
Changing password for user oracle.
New password:oracle
BAD PASSWORD: The password is shorter than 7 characters
Retype new password: oracle
passwd: all authentication tokens updated successfully.

 

현재 사용중인 서비스 확인 및 중지 (RAC1,RAC2)

-서비스확인

[root@ol79rac1 ~]# systemctl list-unit-files | egrep "firewalld.service|chronyd.service|ntpdate.service|avahi-daemon.service"
avahi-daemon.service                          enabled
chronyd.service                               enabled
firewalld.service                             enabled
ntpdate.service                               disabled

- DNS 중지

[root@ol79rac1 ~]# systemctl stop avahi-daemon.socket
[root@ol79rac1 ~]# systemctl stop avahi-daemon
[root@ol79rac1 ~]# systemctl disable avahi-daemon
Removed symlink /etc/systemd/system/multi-user.target.wants/avahi-daemon.service.
Removed symlink /etc/systemd/system/sockets.target.wants/avahi-daemon.socket.
Removed symlink /etc/systemd/system/dbus-org.freedesktop.Avahi.service.

[root@ol79rac1 ~]# systemctl status avahi-daemon
● avahi-daemon.service - Avahi mDNS/DNS-SD Stack
   Loaded: loaded (/usr/lib/systemd/system/avahi-daemon.service; disabled; vendor preset: enabled)
   Active: inactive (dead)

Dec 26 09:27:04 ol79rac1 avahi-daemon[915]: Registering new address record for 192.168.56.101 on enp0s8.IPv4.
Dec 26 09:27:12 ol79rac1 avahi-daemon[915]: Joining mDNS multicast group on interface virbr0.IPv4 with address 192.168.122.1.
Dec 26 09:27:12 ol79rac1 avahi-daemon[915]: New relevant interface virbr0.IPv4 for mDNS.
Dec 26 09:27:12 ol79rac1 avahi-daemon[915]: Registering new address record for 192.168.122.1 on virbr0.IPv4.
Dec 26 09:49:21 ol79rac1 avahi-daemon[915]: Got SIGTERM, quitting.
Dec 26 09:49:21 ol79rac1 systemd[1]: Stopping Avahi mDNS/DNS-SD Stack...
Dec 26 09:49:21 ol79rac1 avahi-daemon[915]: Leaving mDNS multicast group on interface virbr0.IPv4 with address 192.168.122.1.
Dec 26 09:49:21 ol79rac1 avahi-daemon[915]: Leaving mDNS multicast group on interface enp0s8.IPv4 with address 192.168.56.101.
Dec 26 09:49:21 ol79rac1 avahi-daemon[915]: Leaving mDNS multicast group on interface enp0s3.IPv4 with address 10.0.2.10.
Dec 26 09:49:21 ol79rac1 systemd[1]: Stopped Avahi mDNS/DNS-SD Stack.

-시간 동기화 중지

[root@ol79rac1 ~]# systemctl stop chronyd
[root@ol79rac1 ~]# systemctl disable chronyd
Removed symlink /etc/systemd/system/multi-user.target.wants/chronyd.service.

[root@ol79rac1 ~]# systemctl status chronyd
● chronyd.service - NTP client/server
   Loaded: loaded (/usr/lib/systemd/system/chronyd.service; disabled; vendor preset: enabled)
   Active: inactive (dead)
     Docs: man:chronyd(8)
           man:chrony.conf(5)

Dec 26 09:26:50 ol79rac1 chronyd[920]: chronyd version 3.4 starting (+CMDMON +NTP +REFCLOCK +RTC +PRIVDROP +SCFILTER +SIGND +ASYNCDNS +SECHASH +IPV6 +DEBUG)
Dec 26 09:26:50 ol79rac1 chronyd[920]: Frequency -4.331 +/- 0.411 ppm read from /var/lib/chrony/drift
Dec 26 09:26:53 ol79rac1 systemd[1]: Started NTP client/server.
Dec 26 09:27:14 ol79rac1 chronyd[920]: Selected source 211.233.40.78
Dec 26 09:27:14 ol79rac1 chronyd[920]: System clock wrong by -2.213137 seconds, adjustment started
Dec 26 09:27:11 ol79rac1 chronyd[920]: System clock was stepped by -2.213137 seconds
Dec 26 09:28:42 ol79rac1 chronyd[920]: Source 132.226.17.96 replaced with 193.123.243.2
Dec 26 09:38:17 ol79rac1 chronyd[920]: Selected source 193.123.243.2
Dec 26 09:50:07 ol79rac1 systemd[1]: Stopping NTP client/server...
Dec 26 09:50:07 ol79rac1 systemd[1]: Stopped NTP client/server.

-방화벽 중지

[root@ol79rac1 ~]# systemctl stop firewalld
[root@ol79rac1 ~]# systemctl disable firewalld
Removed symlink /etc/systemd/system/multi-user.target.wants/firewalld.service.
Removed symlink /etc/systemd/system/dbus-org.fedoraproject.FirewallD1.service.

[root@ol79rac1 ~]# systemctl status firewalld
● firewalld.service - firewalld - dynamic firewall daemon
   Loaded: loaded (/usr/lib/systemd/system/firewalld.service; disabled; vendor preset: enabled)
   Active: inactive (dead)
     Docs: man:firewalld(1)

Dec 26 09:26:55 ol79rac1 systemd[1]: Starting firewalld - dynamic firewall daemon...
Dec 26 09:27:01 ol79rac1 systemd[1]: Started firewalld - dynamic firewall daemon.
Dec 26 09:27:01 ol79rac1 firewalld[1010]: WARNING: AllowZoneDrifting is enabled. This is considered an insecure configuration option. It will be re...g it now.
Dec 26 09:51:09 ol79rac1 systemd[1]: Stopping firewalld - dynamic firewall daemon...
Dec 26 09:51:10 ol79rac1 systemd[1]: Stopped firewalld - dynamic firewall daemon.
Hint: Some lines were ellipsized, use -l to show in full.

-NTP 중지

[root@ol79rac1 ~]# systemctl stop ntpdate
[root@ol79rac1 ~]# systemctl disable ntpdate

[root@ol79rac1 ~]# systemctl status ntpdate
● ntpdate.service - Set time via NTP
   Loaded: loaded (/usr/lib/systemd/system/ntpdate.service; disabled; vendor preset: disabled)
   Active: inactive (dead)

 

hosts 설정 (RAC1,RAC2)

/etc/hosts 설정

• Public IP : 외부에서 관리자가 접속하는 IP로 관리자가 관리 용도 또는 장애 복구 용도
• Virtual IP : CRS가 Load Balance 할때 쓰는 IP
• Private IP : Node1 - Node2 가 통신할 때만 사용하는 IP로 사용자가 쓰는 것이 아니라 CRS가 instance 끼리 통신하는데 사용하며, 외부에서 접근이 안되기 때문에 보통 사설 IP를 부여한다
• Scan IP : scan ip를 통해오는 접속할 때 하나의 Access name을 가지도록 하는 기능

[root@ol79rac1 ~]# ifconfig
enp0s3: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
내부망  inet 10.0.2.10  netmask 255.255.255.0  broadcast 10.0.2.255
        inet6 fe80::4bee:dfec:2dbd:6f94  prefixlen 64  scopeid 0x20<link>
        ether 08:00:27:25:b6:59  txqueuelen 1000  (Ethernet)
        RX packets 51089  bytes 76064645 (72.5 MiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 5063  bytes 383971 (374.9 KiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

enp0s8: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
외부망  inet 192.168.56.101  netmask 255.255.255.0  broadcast 192.168.56.255
        inet6 fe80::8e97:2840:84bf:6ea7  prefixlen 64  scopeid 0x20<link>
        ether 08:00:27:a4:af:f1  txqueuelen 1000  (Ethernet)
        RX packets 6114  bytes 605688 (591.4 KiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 5652  bytes 1024509 (1000.4 KiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

lo: flags=73<UP,LOOPBACK,RUNNING>  mtu 65536
        inet 127.0.0.1  netmask 255.0.0.0
        inet6 ::1  prefixlen 128  scopeid 0x10<host>
        loop  txqueuelen 1000  (Local Loopback)
        RX packets 6631  bytes 16644295 (15.8 MiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 6631  bytes 16644295 (15.8 MiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

virbr0: flags=4099<UP,BROADCAST,MULTICAST>  mtu 1500
        inet 192.168.122.1  netmask 255.255.255.0  broadcast 192.168.122.255
        ether 52:54:00:43:1a:99  txqueuelen 1000  (Ethernet)
        RX packets 0  bytes 0 (0.0 B)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 0  bytes 0 (0.0 B)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

[root@ol79rac1 ~]# vi /etc/hosts
127.0.0.1   localhost localhost.localdomain localhost4 localhost4.localdomain4
::1         localhost localhost.localdomain localhost6 localhost6.localdomain6

#Public

192.168.56.101 ol79rac1
192.168.56.102 ol79rac2

#Vitual

192.168.56.103 ol79rac1-vip
192.168.56.104 ol79rac2-vip

#Private

10.0.2.10 ol79rac1-priv
10.0.2.11 ol79rac2-priv

#scan

192.168.56.105 ol79rac-scan

 

yum 설정 및 설치 (RAC1,RAC2)

 - 만약 yum repolist가 안된다면 내부 통신망 DNS 설정 확인 필요

[root@ol79rac1 ~]# yum clean all
Loaded plugins: langpacks, ulninfo
Cleaning repos: ol7_UEKR6 ol7_latest

[root@ol79rac1 ~]# yum repolist
Loaded plugins: langpacks, ulninfo
ol7_UEKR6                                                                                                                                                                              | 3.0 kB  00:00:00
ol7_latest                                                                                                                                                                             | 3.6 kB  00:00:00
(1/5): ol7_latest/x86_64/group_gz                                                                                                                                                      | 136 kB  00:00:07
(2/5): ol7_UEKR6/x86_64/updateinfo                                                                                                                                                     | 668 kB  00:00:07
(3/5): ol7_latest/x86_64/updateinfo                                                                                                                                                    | 3.5 MB  00:00:08
(4/5): ol7_UEKR6/x86_64/primary_db                                                                                                                                                     |  48 MB  00:00:12
(5/5): ol7_latest/x86_64/primary_db                                                                                                                                                    |  42 MB  00:00:06
repo id                                                             repo name                                                                                                                           status
ol7_UEKR6/x86_64                                                    Latest Unbreakable Enterprise Kernel Release 6 for Oracle Linux 7Server (x86_64)                                                       816
ol7_latest/x86_64                                                   Oracle Linux 7Server Latest (x86_64)                                                                                                25,007
repolist: 25,823

- 라이브러리 설치 진행

[root@ol79rac1 ~]# vi library.sh

echo '--------------default library--------------'

yum install -y bc
yum install -y binutils
yum install -y compat-libcap1
yum install -y compat-libstdc++-33
yum install -y elfutils-libelf
yum install -y elfutils-libelf-devel
yum install -y fontconfig-devel
yum install -y glibc
yum install -y glibc-devel
yum install -y ksh
yum install -y libaio
yum install -y libaio-devel
yum install -y libX11
yum install -y libXau
yum install -y libXi
yum install -y libXtst
yum install -y libXrender
yum install -y libXrender-devel
yum install -y libgcc
yum install -y libstdc++
yum install -y libstdc++-devel
yum install -y libxcb
yum install -y make
yum install -y net-tools
yum install -y nfs-utils
yum install -y python
yum install -y python-configshell
yum install -y python-rtslib
yum install -y python-six
yum install -y targetcli
yum install -y smartmontools
yum install -y sysstat
yum install -y unzip

echo '--------------Xwindow setting--------------'

yum install -y xorg*

------------------------------------------------------------------------------------------
cf) yum log 확인

[root@ol79rac1 log]# cd /var/log
[root@ol79rac1 log]# ls -al | grep "yum*"
-rw-------.  1 root   root     4114 Dec 26 09:46 yum.log

 

공유 스토리지 생성 및 설정 (이미지는 Virtual box 별로 다를 수 있음)

    - rac1 / rac2 둘다 설정 필요

    - 사용할 스토리지를 지정해서 할당하는 것이 아닌 하나의 디스크를 추가하여 내부적으로 분할 예정

      그렇게 하는 이유는 관리하는 입장에서 편하게 하기 위해 진행

추가로 생성한 dIsk는 공유 가능으로 지정해야 1번 노드 / 2번 노드에서 모두 사용 가능함

 

엔진 설치를 위해 공유 디스크 구성, 파티션 및 파일 시스템 지정 (RAC1)

    - 공유디스크는 /dev/sdb 로 진행

      ( LVM 개념 관련해서는 https://tech.cloud.nongshim.co.kr/2018/11/23/lvmlogical-volume-manager-1-%EA%B0%9C%EB%85%90/)

    - 아래 내용은 rac1 뿐 아니라 rac2번도 적용 필요

LVM(Logical Volumn Manager)

- 디스크 확인
  : fdisk -l
  : fdisk -l /dev/sdb 
  
[root@ol79rac1 ~]# fdisk -l

Disk /dev/sda: 64.4 GB, 64424509440 bytes, 125829120 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk label type: dos
Disk identifier: 0x000b133c

   Device Boot      Start         End      Blocks   Id  System
/dev/sda1   *        2048     2099199     1048576   83  Linux
/dev/sda2         2099200   125829119    61864960   8e  Linux LVM

Disk /dev/sdb: 64.4 GB, 64424509440 bytes, 125829120 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes

Disk /dev/mapper/ol-root: 53.7 GB, 53687091200 bytes, 104857600 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes

Disk /dev/mapper/ol-swap: 9655 MB, 9655287808 bytes, 18857984 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
----------------------------------------------------------------------------------
[root@ol79rac1 ~]# fdisk -l /dev/sdb

Disk /dev/sdb: 64.4 GB, 64424509440 bytes, 125829120 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes

- 디스크 포맷
  : fdisk 디스크명
  : n -> p -> default -> default -> default -> t -> 8e -> p-> w
  
[root@ol79rac1 ~]# fdisk /dev/sdb
Welcome to fdisk (util-linux 2.23.2).

Changes will remain in memory only, until you decide to write them.
Be careful before using the write command.

Device does not contain a recognized partition table
Building a new DOS disklabel with disk identifier 0x7cff3cf3.

Command (m for help): m
Command action
   a   toggle a bootable flag
   b   edit bsd disklabel
   c   toggle the dos compatibility flag
   d   delete a partition
   g   create a new empty GPT partition table
   G   create an IRIX (SGI) partition table
   l   list known partition types
   m   print this menu
   n   add a new partition
   o   create a new empty DOS partition table
   p   print the partition table
   q   quit without saving changes
   s   create a new empty Sun disklabel
   t   change a partition's system id
   u   change display/entry units
   v   verify the partition table
   w   write table to disk and exit
   x   extra functionality (experts only)

Command (m for help): n
Partition type:
   p   primary (0 primary, 0 extended, 4 free)
   e   extended
Select (default p): p
Partition number (1-4, default 1): 1
First sector (2048-125829119, default 2048): 2048
Using default value 2048
Last sector, +sectors or +size{K,M,G} (2048-125829119, default 125829119): 125829119
Using default value 125829119
Partition 1 of type Linux and of size 60 GiB is set

Command (m for help): p

Disk /dev/sdb: 64.4 GB, 64424509440 bytes, 125829120 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk label type: dos
Disk identifier: 0x7cff3cf3

   Device Boot      Start         End      Blocks   Id  System
/dev/sdb1            2048   125829119    62913536   83  Linux


Command (m for help): t
Selected partition 1
Hex code (type L to list all codes): L

 0  Empty           24  NEC DOS         81  Minix / old Lin bf  Solaris
 1  FAT12           27  Hidden NTFS Win 82  Linux swap / So c1  DRDOS/sec (FAT-
 2  XENIX root      39  Plan 9          83  Linux           c4  DRDOS/sec (FAT-
 3  XENIX usr       3c  PartitionMagic  84  OS/2 hidden C:  c6  DRDOS/sec (FAT-
 4  FAT16 <32M      40  Venix 80286     85  Linux extended  c7  Syrinx
 5  Extended        41  PPC PReP Boot   86  NTFS volume set da  Non-FS data
 6  FAT16           42  SFS             87  NTFS volume set db  CP/M / CTOS / .
 7  HPFS/NTFS/exFAT 4d  QNX4.x          88  Linux plaintext de  Dell Utility
 8  AIX             4e  QNX4.x 2nd part 8e  Linux LVM       df  BootIt
 9  AIX bootable    4f  QNX4.x 3rd part 93  Amoeba          e1  DOS access
 a  OS/2 Boot Manag 50  OnTrack DM      94  Amoeba BBT      e3  DOS R/O
 b  W95 FAT32       51  OnTrack DM6 Aux 9f  BSD/OS          e4  SpeedStor
 c  W95 FAT32 (LBA) 52  CP/M            a0  IBM Thinkpad hi eb  BeOS fs
 e  W95 FAT16 (LBA) 53  OnTrack DM6 Aux a5  FreeBSD         ee  GPT
 f  W95 Ext'd (LBA) 54  OnTrackDM6      a6  OpenBSD         ef  EFI (FAT-12/16/
10  OPUS            55  EZ-Drive        a7  NeXTSTEP        f0  Linux/PA-RISC b
11  Hidden FAT12    56  Golden Bow      a8  Darwin UFS      f1  SpeedStor
12  Compaq diagnost 5c  Priam Edisk     a9  NetBSD          f4  SpeedStor
14  Hidden FAT16 <3 61  SpeedStor       ab  Darwin boot     f2  DOS secondary
16  Hidden FAT16    63  GNU HURD or Sys af  HFS / HFS+      fb  VMware VMFS
17  Hidden HPFS/NTF 64  Novell Netware  b7  BSDI fs         fc  VMware VMKCORE
18  AST SmartSleep  65  Novell Netware  b8  BSDI swap       fd  Linux raid auto
1b  Hidden W95 FAT3 70  DiskSecure Mult bb  Boot Wizard hid fe  LANstep
1c  Hidden W95 FAT3 75  PC/IX           be  Solaris boot    ff  BBT
1e  Hidden W95 FAT1 80  Old Minix

Hex code (type L to list all codes): 8e
Changed type of partition 'Linux' to 'Linux LVM'

Command (m for help): p

Disk /dev/sdb: 64.4 GB, 64424509440 bytes, 125829120 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk label type: dos
Disk identifier: 0x7cff3cf3

   Device Boot      Start         End      Blocks   Id  System
/dev/sdb1            2048   125829119    62913536   8e  Linux LVM

Command (m for help): w
The partition table has been altered!

Calling ioctl() to re-read partition table.
Syncing disks.

----------------------------------------------------------------------------

[root@ol79rac1 ~]# fdisk -l /dev/sdb

Disk /dev/sdb: 64.4 GB, 64424509440 bytes, 125829120 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk label type: dos
Disk identifier: 0x7cff3cf3

   Device Boot      Start         End      Blocks   Id  System
/dev/sdb1            2048   125829119    62913536   8e  Linux LVM

- 논리적 스토리지 생성
 pvcreate 디스크명 ▶ vgcreate 그룹명 디스크명 ▶ pvs/vgs로 논리적 스토리지 확인
 (physical volumn)   (volumn group)
 
[root@ol79rac1 ~]# pvcreate /dev/sdb1
  Physical volume "/dev/sdb1" successfully created.

[root@ol79rac1 ~]# vgcreate asm /dev/sdb1
  Volume group "asm" successfully created

[root@ol79rac1 ~]# pvs
  PV         VG  Fmt  Attr PSize   PFree
  /dev/sda2  ol  lvm2 a--  <59.00g   4.00m
  /dev/sdb1  asm lvm2 a--  <60.00g <60.00g

[root@ol79rac1 ~]# vgs
  VG  #PV #LV #SN Attr   VSize   VFree
  asm   1   0   0 wz--n- <60.00g <60.00g
  ol    1   2   0 wz--n- <59.00g   4.00m

- lvs 생성
  : lvcreate -n [볼륨명] -L [사이즈] [volum group명]

[root@ol79rac1 ~]# lvcreate -n mgmt01 -L 40G asm
  Logical volume "mgmt01" created.
[root@ol79rac1 ~]# lvcreate -n orc01 -L 2G asm
  Logical volume "orc01" created.
[root@ol79rac1 ~]# lvcreate -n orc02 -L 2G asm
  Logical volume "orc02" created.

[root@ol79rac1 ~]# lvcreate -n reco01 -L 3G asm
  Logical volume "reco01" created.
[root@ol79rac1 ~]# lvcreate -n reco02 -L 3G asm
  Logical volume "reco02" created.

[root@ol79rac1 ~]# lvcreate -n data01 -L 4.5G asm
  Logical volume "data01" created.
[root@ol79rac1 ~]# lvcreate -n data02 -L 4.5G asm
  Logical volume "data02" created.

[root@ol79rac1 ~]# lvs
  LV     VG  Attr       LSize  Pool Origin Data%  Meta%  Move Log Cpy%Sync Convert
  data01 asm -wi-a-----  4.50g
  data02 asm -wi-a-----  4.50g
  mgmt01 asm -wi-a----- 40.00g
  orc01  asm -wi-a-----  2.00g
  orc02  asm -wi-a-----  2.00g
  reco01 asm -wi-a-----  3.00g
  reco02 asm -wi-a-----  3.00g
  root   ol  -wi-ao---- 50.00g
  swap   ol  -wi-ao----  8.99g
  
-생성된 디스크 확인

[root@ol79rac1 ~]# lsblk
NAME           MAJ:MIN RM  SIZE RO TYPE MOUNTPOINT
sdb              8:16   0   60G  0 disk
└─sdb1           8:17   0   60G  0 part
  ├─asm-data02 252:8    0  4.5G  0 lvm
  ├─asm-reco01 252:6    0    3G  0 lvm
  ├─asm-orc02  252:4    0    2G  0 lvm
  ├─asm-mgmt01 252:2    0   40G  0 lvm
  ├─asm-reco02 252:7    0    3G  0 lvm
  ├─asm-data01 252:5    0  4.5G  0 lvm
  └─asm-orc01  252:3    0    2G  0 lvm
sr0             11:0    1 1024M  0 rom
sda              8:0    0   60G  0 disk
├─sda2           8:2    0   59G  0 part
│ ├─ol-swap    252:1    0    9G  0 lvm  [SWAP]
│ └─ol-root    252:0    0   50G  0 lvm  /
└─sda1           8:1    0    1G  0 part /boot

위 작업을 같이 따라온 상태라면 rac1번에서 생성된 디스크들이 rac2번에는 보이지 않은 현상이 있다. 이는 디스크 내용에 대한 정보가 없어 os 커널이 인식하는 파티션 정보와 일치하지 않기 때문에 발생하는 문제로 reboot 하지 않고 파티션을 인식하는 명령을 통해 진행해야한다.

-rac2번에서 remount 후 추가한 디스크 내역 확인

[root@ol79rac2 ~]# partprobe

[root@ol79rac2 ~]# lvs
  LV     VG  Attr       LSize  Pool Origin Data%  Meta%  Move Log Cpy%Sync Convert
  data01 asm -wi-a-----  4.50g
  data02 asm -wi-a-----  4.50g
  mgmt01 asm -wi-a----- 40.00g
  orc01  asm -wi-a-----  2.00g
  orc02  asm -wi-a-----  2.00g
  reco01 asm -wi-a-----  3.00g
  reco02 asm -wi-a-----  3.00g
  root   ol  -wi-ao---- 50.00g
  swap   ol  -wi-ao----  8.99g

- 디스크 권한 및 소유자 변경을 위한 확인
  (os 부팅을 진행하면 /dm-no 숫자가 변경되기에 하나의 규칙을 만들 필요가 있다)

[root@ol79rac1 asm]# ls -la /dev/asm/*
lrwxrwxrwx. 1 root root 7 Dec 23 16:33 /dev/asm/data01 -> ../dm-5
lrwxrwxrwx. 1 root root 7 Dec 23 16:33 /dev/asm/data02 -> ../dm-8
lrwxrwxrwx. 1 root root 7 Dec 23 16:33 /dev/asm/mgmt01 -> ../dm-2
lrwxrwxrwx. 1 root root 7 Dec 23 16:33 /dev/asm/orc01 -> ../dm-3
lrwxrwxrwx. 1 root root 7 Dec 23 16:33 /dev/asm/orc02 -> ../dm-4
lrwxrwxrwx. 1 root root 7 Dec 23 16:33 /dev/asm/reco01 -> ../dm-6
lrwxrwxrwx. 1 root root 7 Dec 23 16:33 /dev/asm/reco02 -> ../dm-7

udev 설정을 위해서는 유저 생성이 사전에 필요하므로 아래와 같이 유저 생성 필요하다.
(단 공유 스토리지를 설정할 경우 스냅샷을 찍을경우 Virtual box가 깨질수있으니 주의!!)

- udev 설정 
(udev 설명 : https://lascrea.tistory.com/89)

[root@ol79rac1 ~]# cd /etc/udev/rules.d/
[root@ol79rac1 rules.d]# ls -lrt
total 12
-rw-r--r--. 1 root root 628 Sep 26  2019 70-persistent-ipoib.rules
-rw-r--r--. 1 root root 118 May 25  2020 69-vdo-start-by-dev.rules
-rw-r--r--. 1 root root 148 Jun  1  2020 99-vmware-scsi-timeout.rules

[root@ol79rac1 rules.d]# vi 77-raw.rules

KERNEL=="dm-2",OWNER="grid",GROUP="dba",MODE="0660"
KERNEL=="dm-3",OWNER="grid",GROUP="dba",MODE="0660"
KERNEL=="dm-4",OWNER="grid",GROUP="dba",MODE="0660"
KERNEL=="dm-5",OWNER="grid",GROUP="dba",MODE="0660"
KERNEL=="dm-6",OWNER="grid",GROUP="dba",MODE="0660"
KERNEL=="dm-7",OWNER="grid",GROUP="dba",MODE="0660"
KERNEL=="dm-8",OWNER="grid",GROUP="dba",MODE="0660"

[root@ol79rac1 rules.d]# udevadm control --reload-rules

[root@ol79rac1 rules.d]# udevadm trigger --action=add

[root@ol79rac1 rules.d]# ls -l /dev/dm*
brw-rw----. 1 root disk 252, 0 Dec 26 09:34 /dev/dm-0
brw-rw----. 1 root disk 252, 1 Dec 26 09:34 /dev/dm-1
brw-rw----. 1 grid dba  252, 2 Dec 26 09:34 /dev/dm-2
brw-rw----. 1 grid dba  252, 3 Dec 26 09:34 /dev/dm-3
brw-rw----. 1 grid dba  252, 4 Dec 26 09:34 /dev/dm-4
brw-rw----. 1 grid dba  252, 5 Dec 26 09:34 /dev/dm-5
brw-rw----. 1 grid dba  252, 6 Dec 26 09:34 /dev/dm-6
brw-rw----. 1 grid dba  252, 7 Dec 26 09:34 /dev/dm-7
brw-rw----. 1 grid dba  252, 8 Dec 26 09:34 /dev/dm-8

---

디렉토리 생성 및 profile 설정 (RAC1,RAC2)

1. 환경설정

- rac1번 os grid의 bash_profile

# Grid Environment
export ORACLE_BASE=/grid_engine/base
export ORACLE_HOME=/grid_engine/12c/grid
export GRID_HOME=$ORACLE_HOME
export ORACLE_SID=+ASM1
export BASE_PATH=/usr/sbin
export PATH=$PATH:$ORACLE_HOME/bin:$ORACLE_HOME/OPatch:$GRID_HOME/bin:$BASE_PATH:$HOME/.local/bin:$HOME/bin
export LD_LIBRARY_PATH=$ORACLE_HOME/lib:/lib:/usr/lib
export CLASSPATH=$ORACLE_HOME/JRE:$ORACLE_HOME/jlib:$ORACLE_HOME/rdbms/jlib
 
export PS1=`whoami`@`hostname`\['$ORACLE_SID'\]':$PWD$ '

- rac1번 os oracle의 bash_profile

# Oracle Environment:
export ORACLE_BASE=/ora_engine
export ORACLE_HOME=$ORACLE_BASE/12c/dbhome_1
export ORACLE_PATH=$ORACLE_HOME/bin:$ORACLE_HOME/OPatch
export ORACLE_SID=RACDB1
export TNS_ADMIN=$ORACLE_HOME/network/admin
export NLS_LANG=AMERICAN_AMERICA.KO16MSWIN949
export CLASSPATH=$ORACLE_HOME/JRE:$ORACLE_HOME/jlib:$ORACLE_HOME/rdbms/jlib:$ORACLE_HOME/network/jlib
export PATH=$PATH:$ORACLE_PATH:$HOME/.local/bin:$HOME/bin
 
# Set up the shell variables:
export EDITOR=vi
export LANG=C
export PS1=`whoami`@`hostname`\['$ORACLE_SID'\]':$PWD$ '

-----------------------------------------------------------------------------------

- rac2번 os grid의 bash_profile

# Grid Environment
export ORACLE_BASE=/grid_engine/base
export ORACLE_HOME=/grid_engine/12c/grid
export GRID_HOME=$ORACLE_HOME
export ORACLE_SID=+ASM2
export BASE_PATH=/usr/sbin
export PATH=$PATH:$ORACLE_HOME/bin:$ORACLE_HOME/OPatch:$GRID_HOME/bin:$BASE_PATH:$HOME/.local/bin:$HOME/bin
export LD_LIBRARY_PATH=$ORACLE_HOME/lib:/lib:/usr/lib
export CLASSPATH=$ORACLE_HOME/JRE:$ORACLE_HOME/jlib:$ORACLE_HOME/rdbms/jlib
 
export PS1=`whoami`@`hostname`\['$ORACLE_SID'\]':$PWD$ '

- rac2번 os oracle의 bash_profile

# Oracle Environment:
export ORACLE_BASE=/ora_engine
export ORACLE_HOME=$ORACLE_BASE/12c/dbhome_1
export ORACLE_PATH=$ORACLE_HOME/bin:$ORACLE_HOME/OPatch
export ORACLE_SID=RACDB2
export TNS_ADMIN=$ORACLE_HOME/network/admin
export NLS_LANG=AMERICAN_AMERICA.KO16MSWIN949
export CLASSPATH=$ORACLE_HOME/JRE:$ORACLE_HOME/jlib:$ORACLE_HOME/rdbms/jlib:$ORACLE_HOME/network/jlib
export PATH=$PATH:$ORACLE_PATH:$HOME/.local/bin:$HOME/bin
 
# Set up the shell variables:
export EDITOR=vi
export LANG=C
export PS1=`whoami`@`hostname`\['$ORACLE_SID'\]':$PWD$ '

2. 디렉토리 생성 및 권한 설정

- rac1번

[root@ol79rac1 ~]# mkdir -p /grid_engine/base
[root@ol79rac1 ~]# mkdir -p /grid_engine/12c/grid
[root@ol79rac1 ~]# mkdir -p /ora_engine/12c/dbhome_1

[root@ol79rac1 ~]# chown -R grid. /grid_engine
[root@ol79rac1 ~]# chown -R oracle. /ora_engine

- rac2번

[root@ol79rac1 ~]# mkdir -p /grid_engine/base
[root@ol79rac2 ~]# mkdir -p /grid_engine/12c/grid
[root@ol79rac2 ~]# mkdir -p /ora_engine/12c/dbhome_1

[root@ol79rac2 ~]# chown -R grid. /grid_engine
[root@ol79rac2 ~]# chown -R oracle. /ora_engine

 

오라클 설치를 위한 커널 파라미터 설정 (RAC1,RAC2)

https://docs.oracle.com/en/database/oracle/oracle-database/12.2/ladbi/changing-kernel-parameter-values.html#GUID-FB0CC366-61C9-4AA2-9BE7-233EB6810A31

 

Installation Guide

https://redkite777.tistory.com/entry/%EC%98%A4%EB%9D%BC%ED%81%B4%EC%84%A4%EC%B9%98-%EC%8B%9C-%EC%BB%A4%EB%84%90-%EB%A7%A4%EA%B0%9C%EB%B3%80%EC%88%98-%EC%84%A4%EC%A0%95

[오라클]설치 시 커널 매개변수 설정

[root@ol79rac1 ~]# vi /etc/sysctl.conf

kernel.sem = 250 32000 100 128
kernel.shmmni = 4096
kernel.shmall = 2549152         # MemTotal (KB) * 1024 / 4096    page 단위 (Doc ID 301830.1)
kernel.shmmax = 5220685824      # MemTotal (KB) * 1024 / 2       bytes 단위(Doc ID 567506.1)
kernel.panic_on_oops = 1

net.core.rmem_default = 262144
net.core.rmem_max = 4194304
net.core.wmem_default = 262144
net.core.wmem_max = 1048576

net.ipv4.conf.all.rp_filter = 2
net.ipv4.conf.default.rp_filter = 2

fs.file-max = 6815744
fs.aio-max-nr = 1048576

net.ipv4.ip_local_port_range = 9000 65500

※ Kernel.shmmax : 공유 meory segment의 max 크기(byte)를 정의하는데 사용하며 오라클 SGA 메모리로 구성되며, SHMMAX가 올바르게 설정되지 않을 경우 SGA의 크기가 제약될 수 있다.만약 잘못 설정될 경우 아래와 같은 에러 발생
'ORA-27123 : unable to attach to shared memory segment' 

※ Kernel.shmall : 특정 시점에 시스템에서 사용 가능한 공유 memory의 최대 크기(page)를 설정하는데 사용하며, celi(shmmax/page_size)보다 커야함.
 - Linux page size 확인 getconf PAGESIZE |sed -e :a -e 's/\(.*[0-9]\)\([0-9]\{3\}\)/\1.\2/;ta'
 - shmall default size : getconf PAGESIZE |sed -e :a -e 's/\(.*[0-9]\)\([0-9]\{3\}\)/\1.\2/;ta'

※ Kernel.sem(=semaphore) : 공유 리소스의 사용 과정에서 프로세스 간의 동기화를 위해 사용되는 일종의 카운터
- semaphore 설정값 확인
 ipcs -lm
------ Shared Memory Limits --------
max number of segments = 4096
max seg size (kbytes) = 5098304
max total shared memory (kbytes) = 10196608
min seg size (bytes) = 1

- oracle에서 세마포어 최대값 조정
cat /proc/sys/kernel/sem |sed -e :a -e 's/\(.*[0-9]\)\([0-9]\{3\}\)/\1.\2/;ta'
250 32000 100 128

 

유저리소스제한 (RAC1,RAC2),(rac2번에 적용할 경우 racdb1 > racdb2로 변경 후 적용)

Linux 계정 별로 실행되는 프로세스와 열린 파일 수를 제한 하는 것을 권장한다. 

- nproc : 프로세스 개수

- nofile : 한번에 열 수 있는 최대 파일 수

 - soft limit : 새로운 프로그램이 생성되면 default로 적용되는 제한

 - hard limit : soft limit에서 최대 늘릴 수 있는 제한

                  (단, hard limit은 root에 의해 조정이 가능하지만 무한으로 늘릴 수 없어서 kernel에서 해당 설정값 조정 후 가)

[root@ol79rac1 etc]# vi /etc/security/limits.conf

grid    soft   nproc     2047
grid    hard   nproc    16384
grid    soft   nofile     1024
grid    hard   nofile    65536
grid    soft   stack     10240
grid    hard   stack    32768

oracle   soft    nofile      1024
oracle   hard   nofile       65536
oracle   soft    nproc      16384
oracle   hard   nproc      16384
oracle   soft    stack       10240
oracle   hard   stack       32768
oracle   soft    memlock   9176947 # MemTotal의 약 90% (KB)
oracle   hard   memlock   9176947  # MemTotal의 약 90% (KB)