Frequency Response Analysis

s-plane contour is encircling 2-poles and 3-zeros in clockwise direction hence the corresponding G(s) plane contour encircles origin 2-times in anti-clockwise direction and 3-times in clockwise direction.
Therefore, Effectively once in clockwise direction.

Number of poles (N_{P})= 6
Number of zeros (N_{Z}) = 3

For G(s)
M_{\mathrm{dB}}\left(\omega_{p c}\right)=20 \mathrm{dB}
When cascaded with k ,
\begin{aligned} \mathrm{GM}_{\mathrm{dB}} &=-20 \mathrm{dB}-20 \log _{10}(k) \gt 0 \mathrm{dB} \\ 20+20 \log _{10}(k) & \lt 0 \\ 20 \log _{10}(k) & \lt -20 \\ k & \lt 10^{-1}=0.10\\ \text{So,}\quad k_{0}&=0.10 \end{aligned}

Maximum resonant peak,
\begin{aligned} M_{r} &=\frac{1}{2 \xi \sqrt{1-\xi^{2}}}=2 \\ 2 \xi \sqrt{1-\xi^{2}} &=\frac{1}{2} \\ \xi^{2}\left(1-\xi^{2}\right) &=\frac{1}{16} \\ \xi^{4}-\xi^{2}+\frac{1}{16} &=0 \\ \xi^{2}&=\frac{1}{2} \pm \sqrt{\frac{1-\frac{1}{4}}{4}}=\frac{1}{2} \pm \frac{\sqrt{3}}{4} \\ \text { As } M_{r}=2 \gt 1, \xi& \lt \frac{1}{\sqrt{2}} \text { and } \xi^{2} \lt \frac{1}{2} \\ \text { So. } \quad\xi^{2}&=\frac{1}{2}-\frac{\sqrt{3}}{4}\\ Given, \quad G(s)&=\frac{K}{s(s+2)}=\frac{\omega_{n}^{2}}{s\left(s+2 \xi \omega_{n}\right)} \\ \text{So} \quad\omega_{n} &=\sqrt{K} \\ 2 \xi \sqrt{K} &=2\\ \sqrt{K} &=\frac{1}{\xi} \\ K &=\frac{1}{\xi^{2}}=\frac{1}{\left(\frac{1}{2}-\frac{\sqrt{3}}{4}\right)} \\ &=\frac{4}{2-\sqrt{3}}=14.928 \end{aligned}

Given that, the open loop transfer function of a unity feedback system is,
G(s) =\frac{10 k(s+2)}{s^{3}+3 s^{2}+10}
From the given Nyquist plot, for 0 \lt k \lt 1 the encirclements about the point (-1+j 0) is,
\begin{array}{l} N=0 \\ N=P-Z \end{array}
P= number of open loop poles in the right half of s-plane
Z= number of closed loop poles in the right half of s-plane
To determine the value of P :
Applying R-H criteria to G(s)
G(s)=\frac{10 k(s+2)}{s^{3}+3 s^{2}+10}
\begin{array}{c|cc} s^{3} & 1 & 0 \\ s^{2} & 3 & 10 \\ s^{1} & -10 / 3 & 0 \\ s^{0} & 10 & 0 \end{array}
Two sign changes are there in the first column of the RH table. So, two open loop poles are there in right half of s-plane.
So,
P=2
\begin{array}{l} N=P-Z \\ 0=2-Z \\ Z=2 \end{array}
Z=2 indicates, there are two closed loop poles in right half of s-plane.

Given that,
G(s)=\frac{K}{\left(s^{2}+2 s+2\right)(s+2)}
P= open loop poles in RHS of s-plane
Z= closed loop poles in RHS of s-plane
N= number of encirclements about the
\begin{array}{l} \text { point }(-1+j 0) \\ \text { For } K=10 ; \\ N=P-Z=0 \Rightarrow Z=0: \text { stable } \\ \text { For } K=100 ; \\ N=P-Z=1 \Rightarrow Z \neq 0: \text { unstable } \end{array}

Final slope = -60 dB/decade, which indicates that, q-p=3
Among the given options, option (A) satisfies this condition

\begin{aligned} G(s)=\frac{K}{(s+0.1)(s+10)\left(s+p_{1}\right)} \\ \angle G(s)=-\tan ^{-1} \frac{\omega}{0.1}-\tan ^{-1} \frac{\omega}{10}-\tan ^{-1} \frac{\omega}{p_{1}} \\ \left.\angle G(s)\right|_{\omega=1}=-\tan ^{-1} \frac{1}{0.1}-\tan ^{-1} \frac{1}{10}-\tan ^{-1} \frac{1}{p_{1}} \\ =-135^{\circ} \\ -\tan ^{-1} 10-\tan ^{-1} 0.1-\tan ^{-1} \frac{1}{p_{1}}=-135^{\circ} \\ -84.28^{\circ}-5.71^{\circ}-\tan ^{-1} \frac{1}{p_{1}}=-135^{\circ} \\ -\tan ^{-1} \frac{1}{p_{1}}=-135^{\circ}+90^{\circ}\\ \tan ^{-1} \frac{1}{p_{1}} =45^{\circ} \\ \frac{1}{p_{1}} =1 \Rightarrow p_{1}=1 \end{aligned}

1+G(s) H(s)=1+\frac{K}{(s+1)(s+2)(s+3)}=0
(s+1)\left(s^{2}+5 s+6\right)+K=0
s^{3}+5 s^{2}+6 s+s^{2}+5 s+6+k=0
\Rightarrow s^{3}+6 s^{2}+11 s+6+K=
Gain margin =0dB and phase margin =0^{\circ}
It implies marginal stable system
By Routh Array
\begin{array}{r|rr} s^{3} & 1 & 11 \\ s^{2} & 6 &(6+k) \\ s & \frac{66-6-K}{6} & 0 \\ s^{\circ} & 6+K & \end{array}
For marginal stable system,
60-K=0
K=60