Evaluation of Accuracy of Microplate Alamar Blue Assay and Proportion Method for Prompt Detection of Mycobacterium tuberculosis and Clinical Isolates of Multidrug-resistant M. tuberculosis

AUTHORS

Alireza Jafari ORCID 1 , * , Raj Goswami 2 , Hesamaddin Shirzad Aski 3 , Nasser Behnampour 4 , Masoomeh Taziki 3 , Masoomeh Atarjalali 3 , Elahe Rafiee 5

1 Urology Research Center, Razi Hospital, School of Medicine, Guilan University of Medical Sciences, Rasht, Iran

2 Institute for Tuberculosis Research, College of Pharmacy, University of Illinois at Chicago, Chicago, Illinois, USA

3 Infectious Diseases Research Center, Golestan University of Medical Sciences, Gorgan, Iran

4 Golestan University of Medical Sciences, Gorgan, Iran

5 Guilan University of Medical Sciences, Rasht, Iran

How to Cite: Jafari A, Goswami R , Shirzad Aski H , Behnampour N , Taziki M, et al. Evaluation of Accuracy of Microplate Alamar Blue Assay and Proportion Method for Prompt Detection of Mycobacterium tuberculosis and Clinical Isolates of Multidrug-resistant M. tuberculosis. Jundishapur J Microbiol. 2021;14(3):e111212. doi: 10.5812/jjm.111212.

ARTICLE INFORMATION

Jundishapur Journal of Microbiology: 14 (3); e111212
Published Online: August 1, 2021
Article Type: Research Article
Received: February 18, 2021
Revised: April 24, 2021
Accepted: April 28, 2021
Crossmark
Crossmark
CHECKING
READ FULL TEXT

Abstract

Background: Tuberculosis is appraised to cause the deaths of more than a billion people in the last decades.

Objectives: The current study compares the performance of microplate Alamar blue assay for clinical isolates of Mycobacterium tuberculosis and multidrug-resistant tuberculosis. Microplate Alamar blue assay was performed in a central tuberculosis laboratory at Golestan University of Medical Sciences in Gorgan, Iran.

Methods: In the first step, the microplate Alamar blue assay was used for the detection of 78 clinical isolates in the Golestan regional tuberculosis reference laboratory, and the results were compared with those of the proportion assay. In the second step, the microplate Alamar blue assay and the proportion assay were used for the drug-susceptibility of 35 isolates.

Results: In the microplate Alamar blue assay, the sensitivity was 100 (90.97 - 100), with a specificity of 74.36 (57.87 - 86.96), positive predictive value of 79.59 (65.66 - 89.76), and negative predictive value of 100 (88.06 - 100). For the microplate Alamar blue assay with rifampin, the sensitivity was 100 (89.11 - 100), specificity was 100 (29.24 - 100), positive predictive value was 100 (89.11 - 100), and negative predictive value was 100 (29.24 - 100). For the microplate Alamar blue assay with isoniazid, the sensitivity was 84.38 (67.21 - 94.72), specificity was 66.67 (9.43 - 99.16), positive predictive value was 96.43 (81.65 - 99.91), and negative predictive value was 28.57 (3.67 - 70.96).

Conclusions: We found high accuracy between the microplate Alamar blue assay with rifampin and the proportion assay. The rapid and low-cost microplate Alamar blue assay is an inexpensive and appropriate assay for the detection of rifampin-resistant tuberculosis in low-income countries.

1. Background

Tuberculosis is appraised to have caused the deaths of more than a billion people in the last decades (1). Multidrug-resistant Mycobacterium tuberculosis has been converted to an important problem for tuberculosis control because patients rarely have responded to anti-tuberculosis therapy (2). Hence, the simple, low-cost, and prompt detection of multidrug-resistant M. tuberculosis can prevent the spread of the disease and help to select appropriate treatments (3). In many countries, tuberculosis resistance to antibiotics often goes undetected and untreated. Most national tuberculosis programs in developing countries do not contain culture and drug-susceptibility testing. The early choice of appropriate treatment is an essential determinant of the reduced morbidity, mortality, and infectiousness (4).

On the Lowenstein-Jensen medium under the proportion assay, it takes up to 12 weeks to identify drug-resistant tuberculosis. Taking a long time to identify drug-resistant tuberculosis with expensive and advanced equipment not only has been a challenge not only for patients, but also for governments. As mentioned before, within weeks of starting tuberculosis, an inappropriate treatment regimen may result in death (4). However, the liquid culture systems significantly reduce turnaround time. But with liquid cultures, it takes two to four weeks to detect drug resistance, which substantially imposes high costs on patients in developing countries (4). Todays, oxidation-reduction dyes are used to obtain drug susceptibility of mycobacteria (5, 6). Alamar blue is a resazurin-based oxidation-reduction indicator. It is able to deliver the colorimetric minimum inhibitory concentration (MIC) for M. tuberculosis for two weeks (7). In this respect, Yajko et al. reported a good correlation between the proportion assay and Alamar blue (6). Also, Franzblau et al. showed that the microplate Alamar blue assay with modified medium composition could reduce reaction time (1).

2. Objectives

The current study compared the performance of the microplate Alamar blue assay for clinical M. tuberculosis and multidrug-resistant tuberculosis isolates. The microplate Alamar blue assay was performed in a central tuberculosis laboratory at Golestan University of Medical Sciences in Gorgan, Iran (8, 9).

3. Methods

3.1. Patients

Seventy-eight sputum samples with tuberculosis positive patients (group one) and 39 sputum samples with tuberculosis negative patients (group two) who were referred to Golestan regional tuberculosis laboratory were studied. Samples of non-tuberculosis were excluded from the study. The required sample size was estimated at 78 patients (39 patients in group one and 39 patients in group two) based on the sensitivity values and characteristics of the Alamar blue assay in the study by Farnia et al. (10) with 95% confidence.

3.2. Strains and Drug Preparation

Seventy-eight M. tuberculosis samples were tested including multidrug-resistant isolates in addition to H37Rv, ATCC 35822 (Isoniazid resistant), and ATCC 35838 (Rifampin resistant) standard strains (American Type Culture Collection, Rockville, Md.). Mycobacterium tuberculosis samples were obtained from Golestan regional tuberculosis reference laboratory at Golestan University of Medical Sciences (Gorgan, Iran), and the drug susceptibility profiles of these samples were obtained using the proportion assay. Twenty-seven multidrug-resistant tuberculosis strains were resistant to isoniazid, and 32 multidrug-resistant tuberculosis strains were resistant to rifampin. The multidrug-resistant strains were sub-cultured on Middle brook 7H11 agar (Becton Dickinson Microbiology Systems, Cockeysville, Md.). Then, the suspensions were prepared in 0.04% (v/v) Tween 80 - 0.2% bovine serum albumin (Sigma Chemical Co., St. Louis, Mo.) with No.1 standard turbidity of McFarland (6).

Samples were diluted at a ratio of 1:25 in 7H9GC broth (4.7 g of Middle brook 7H9 broth [Remel, Lenexa, Kans.]). A stock solution of isoniazid was prepared in deionized water, and the stock solution rifampin was prepared in dimethyl sulfoxide. Stock solutions (2X concentration) were diluted in 7H9GC broth (1). All samples were digested and disinfected using acetylcysteine and sodium hydroxide at the concentrations of 2% and 5%, respectively (11). After about 15 min, the samples were centrifuged at 3,000 g for 30 min, and 1 to 2 mL of the remaining precipitate were removed. They were then neutralized by a few drops of hydrochloric acid and flushed with phosphate-buffered saline (0.067 mol at pH 6.8). After that, 2 mL of penicillin containing phosphate-buffered saline was added to the remaining sediment, and then 250 L of the suspension was inoculated into the Lowenstein-Jensen medium (9). Catalase, niacin, nitrate, and thiophene 2-carboxylic acid hydrazide tests were used to diagnose Mycobacterium species.

3.3. Diagnosis of Mycobacterium in Sputum by Alamar Blue Assay

A 7H9GC broth medium was prepared and transferred to 15 mL Falcon tubes. A 7H9GC drug-free broth solution was used to control culture medium. Then, 250 µL were added to the tubes from the freshly prepared mixture of Almar Blue 10' and 10% Tween 80 at a ratio of 1 to 1. In all test tubes, 0.1 mL of sputum suspension was inoculated. The tubes were incubated at 37ºC for four weeks, and the color change of the medium was recorded daily. Color change from blue to pink indicated bacterial growth in drug-containing tubes, and the absence of bacteria was indicated if the tubes did not change color. In the control group, 7H9GC broth and bacteria were added to 250 μL of Almar Blue (8).

3.4. Diagnosis of Mycobacterium in Sputum Specimen by Microplate Alamar Blue

About 200 µL of distilled water was added twice to each well of a 96-well plate, and then about 100 mL of 7H9GC broth was added to each well. About 100 µL of sputum suspension was added to the wells. The last wells were considered as controlled group because they did not have a sample of sputum suspension. The microplates were sealed with parafilm and incubated for 37 days in a 37°C incubator. On the fifth day, about 50 L of Alamar Blue 10 and Tween 80 were added to the first well at a ratio of 1:1, and were incubated at 37°C for 24 h. If the first well-turned pink, the compound would be added to all wells. If the blue well remained, the compound would be added to the next control well, and the results would be followed the next day. The microplate was sealed with parafilm and was incubated at 37°C for another 24 hours. During this time, changes in the color of the well were investigated. The blue color in the well was a sign of bacterial growth, and the pink color was a sign of bacterial growth. Few wells turned purple after 24 hours but turned pink after 24 hours later (10).

3.5. Determination of Drug Susceptibility to Rifampin and Isoniazid with Proportion Assay

Nowadays, the proportional assay is still used to determine the drug susceptibility of Mycobacterium to antitubercular drugs. This method is standard and approved by the World Health Organization, Clinical and Laboratory Standards Institute. This procedure requires at least 3 - 4 weeks of incubation (1). For this purpose, a diluted bacterial suspension of 0.5 McFarland was first prepared for each sample of positive smear patients. Thus, some bacterial colonies were removed from the original culture medium by sterile plastic spatulas and transferred to the Mc Carteny tube, which contained 20 - 30 - 20 sterile glass pearls. It was then sealed in a tight tube and vortexed for 15 seconds.

Sterile distilled water was added to it by a Pasteur pipette, and it was mixed again for 15 seconds after the lid was firmly closed. From the obtained suspension, 0.5 McFarland was prepared, and 0.001 M. tuberculosis suspension was prepared, furthermore, 0.1 mL dilution of 0.5 McFarland was added to 9.9 mL of sterile distilled water. Then, 0.2 mL of 0.01 diluted suspension was inoculated into each tube. Löwenstein-Jensen antibiotic-containing culture media and the antibiotic-free culture medium (control culture medium). The cultured media were incubated at 37ºC for 48 - 42 days, and daily changes were controlled (1). According to the standards, the final concentration of rifampin in the culture medium was 40 μg/mL, and the final concentration of isoniazid was 0.2 μg/mL. After incubation, the number of colonies grown between the controlled medium and the antibiotic-containing medium were compared.

3.6. Determination of Drug Susceptibility to Rifampin and Isoniazid by Alamar Blue Assay

The antifungal dilutions of rifampin 2 µg/mL and isoniazid 0.2 µg/mL were prepared in a 7H9GC broth medium and then 1.65 mL was transferred to 15 mL Falcon tubes. A 7H9GC drug-free broth solution was used to control the bacteria. Then, 250 µL of freshly prepared mixture of Almar Blue 10 and 10% Tween 80 was added to the tubes containing antibiotics and control (no antibiotics) at a ratio of 1 to 1. In all test tubes, 0.1 mL of bacterial dilution was incubated at 37ºC for four weeks, and the color change of the medium was under observation daily. The tubes in which color changed from blue to pink had anti-bacterial resistance, and the tubes which remained blue had susceptibility to antibiotics. In the control groups, 250 g of Almar Blue were added to the 7H9GC broth containing antibiotics (Negative control), and the 7H9GC broth without antibiotics (Positive control) (8).

3.7. Determination of Drug Susceptibility to Rifampin and Isoniazid by Microplate Alamar Blue Assay

About 200 μL of distilled water was added twice to each well of a 96-well plate, and then about 100 mL of the 7H9GC broth culture medium was added to each well in rows B and G in columns 3 to 11. To the wells of columns 2 and 3 in rows B to G, about 100 μL of the drug solution was added. Using a multi-channel pipette, 100 μL of the contents of column 3 was transferred to column 4. Serial dilutions continued until column 10, and eventually 100 μL was discarded from column 10. About 100 μL of the M. tuberculosis suspension was added to wells (B to G) in columns 2 to 11. Wells in row 11 were considered as control which were without drugs. The microplates were sealed with parafilm and incubated in a 37ºC incubator for five days. About 50 L of the Alamar Blue and Tween 80% was added to the wells B11 (ratio of 1:1), and microplates were incubated at 37ºC for 24 hours. The microplate was sealed with parafilm and incubated for another 24 h at 37ºC.

During this time, changes in the color of the wells were under observation. The wells which turned blue indicated no bacterial growth and the wells with pink were a sign of bacterial growth. Few other wells turned pink after 24 hours. The amount of MIC was the lowest concentration of the drug that prevented changes from blue to pink (10). Sensitivity and specificity are statistical measures widely used in basic sciences and medical research. Sensitivity is the true positive rate defined as the proportion of positives that are correctly identified. Specificity is the true negative rate defined as the proportion of negatives that are correctly identified.

3.8. Statistical Analysis

All data were analyzed by SPSS 18 software, and the results of the Alamar Blue assay and microplate Alamar blue assay were compared with those of the proportion assay. The agreement between Alamar Blue and microplate Alamar blue assay was measured through the Kappa test (12).

4. Results

The sensitivity of the microplate Alamar blue assay test with rifampin was calculated as 100 (89.11 - 100), with a specificity of 100 (29.24 - 100), positive predictive value of 100 (89.11 - 100), and negative predictive value of 100 (29.24 - 100). The sensitivity of the microplate Alamar blue assay with isoniazid was estimated at 84.38 (67.21 - 94.72), with a specificity of 66.67 (9.43 - 99.16), positive predictive value of 96.43 (81.65 - 99.91), and negative predictive value of 28.57 (3.67 - 70.96) (Tables 1 and 2).

Table 1. Diagnostic Value Indicators of Alamar Blue and Microplate Alamar Blue Methods Compared to Proportion Method
IndexMicroplate Alamar Blue Assay % (95%CI)Alamar Blue % (95%CI)
Sensitivity100 (90.97 - 100)74.36 (57.87 - 86.96)
Specificity74.36 (57.87 - 86.96)58.97 (42.10 - 74.43)
Positive predictive value79.59 (65.66 - 89.76)64.44 (48.78 - 78.13)
Negative predictive value100 (88.06 - 100)69.70 (51.29 - 84.41)

Abbreviation: CI, confidence interval.

Table 2. Diagnostic Value Indicators of Microplate Alamar Blue Assay Method in Relation to Susceptibility to Rifampin and Isoniazid
IndexSusceptibility to Isoniazid % (95%CI)Susceptibility to Rifampin % (95%CI)
Sensitivity84.38 (67.21 - 94.72)100.00 (89.11 - 100.00)
Specificity66.67 (9.43 - 99.16)100.00 (29.24 - 100.00)
Positive predictive value96.43 (81.65 - 99.91)100.00 (89.11 - 100.00)
Negative predictive value28.57 (3.67 - 70.96)100.00 (29.24 - 100.00)

Abbreviation: CI, confidence interval.

5. Discussion

Nowadays, the proportion method is a routine method to diagnose and determine antibiotic susceptibility to M. tuberculosis. This method requires a lot of time, which is commonly used in most tuberculosis diagnostic laboratories, especially in developing countries. Although this method is more acceptable, it also has disadvantages, as it is very time consuming and laborious. To evaluate drug sensitivity by this method, first, it is necessary to cultivate, isolate, and then perform antibiotic sensitivity. According to the above steps, it needs a lot of time, about 8 to 10 weeks. Rapid methods include radiometric methods and molecular methods, require expensive equipment. Therefore, to diagnose and determination of antibiotic susceptibility of M. tuberculosis, it is necessary to use methods that are cheaper and fast.

The micro-dilution assays are cost-effective, rapid, and with high quality in comparison with phenotypic drug susceptibility testing. In this study, the microplate Alamar blue assay was used for the detection of 78 clinical isolates from Golestan regional tuberculosis reference laboratory, and the results were compared with those of the proportion method. Also, the microplate Alamar blue assay and the proportion method were used for drug susceptibility testing of 35 isolates. Our findings showed that the sensitivity and specificity of microplate Alamar blue assay were 100 (90.97 - 100) and 74.36 (57.87 - 86.96). Also, we calculated the positive predictive value and negative predictive value of the microplate Alamar blue assay as 79.59 (65.66 - 89.76) and 100 (88.06 - 100). The sensitivity of the microplate Alamar blue assay with rifampin was calculated as 100 (89.11 - 100), with a specificity of 100 (29.24 - 100), positive predictive value of 100 (89.11 - 100), and negative predictive value of 100 (29.24 - 100). The sensitivity of the microplate Alamar blue assay with isoniazid was estimated at 84.38 (67.21 - 94.72), with a specificity of 66.67 (9.43 - 99.16), positive predictive value of 96.43 (81.65 - 99.91), and negative predictive value of 28.57 (3.67 - 70.96).

Many studies reported that the microplate Alamar blue assay seems to be an excellent alternative to phenotypic drug susceptibility testing. Farnia used the blue test in 2003 to evaluate the viability of M. tuberculosis in patients' sputum samples (10). For the determination of antibiotic susceptibility of M. tuberculosis, Caviedes et al. (11) used the kind of colorimetric method named "Tetrazolium Microplate Assay," which was rapid and inexpensive even than microplate Alamar Blue Assay. Caviedes et al. (11) showed that the colorimetric method could provide acceptable results within six to seven days. To evaluate the drug susceptibility of bacillus tuberculosis, Caviedes et al. (11) used the microplate Alamar blue assay and showed that this method could provide acceptable results within six to seven days. Franzblau et al. (1) proposed that the microplate Alamar blue assay could be used to determine the minimum inhibitory concentration of antitubercular drugs. Franzblau et al. obtained the MIC results for all samples within eight days (1). They showed that the agreement between BACTEC and microplate Alamar blue assay was more than 87% in accurately determining the MIC, which was an acceptable result. They reported that the microplate Alamar blue assay could be used as an easy and low-cost method in developing countries.

Our study showed that the sensitivity and specificity of the microplate Alamar blue assay were 100 and 74.36, respectively. The sensitivity and specificity of microplate Alamar blue assay with rifampin were 100 and 100, respectively. The sensitivity and specificity were 84.38 and 66.67, respectively, in the microplate Alamar blue assay with isoniazid. In this respect, the results of our study were inconsistent with those of Chauca et al. (2), in which the sensitivity and specificity of microplate Alamar blue assay in drug susceptibility testing of 63 isolates of M. tuberculosis to rifampin and isoniazid were 98% and 97%, respectively. Our study showed that the positive predictive value and negative predictive value for the microplate Alamar blue assay test were 79.59 and 100, respectively. For the microplate Alamar blue assay with rifampin, the positive predictive value and negative predictive value were 100 and 100, respectively. The negative predictive value was 28.57. Chauca et al. (2) showed that the negative predictive value and positive predictive value of microplate Alamar blue assay in the drug susceptibility testing of 63 isolates of M. tuberculosis against rifampin were 99.9 and 80.3, respectively.

Previous studies showed that the negative predictive value and positive predictive value of microplate Alamar blue assay in drug susceptibility testing of 63 isolates of M. tuberculosis against isoniazid were 99.7 and 76.2, respectively. Based on current research, high accuracy was observed between the microplate Alamar blue assay with rifampin and the proportion method. We found that the microplate Alamar blue assay could have greater diagnostic accuracy than the gold standard (the proportion method) on the Löwenstein-Jensen medium. It should be noted that the microplate Alamar blue assay in the microplate format is best suited for tuberculosis reference laboratories. To determining the susceptibility of rifampin-resistant M. tuberculosis, the agreement between the microplate Alamar Blue assay and the proportion method was estimated at 100%. The mean time for determining susceptibility to rifampin-resistant M. tuberculosis was six days. While in the proportion method, the duration of susceptibility determination of rifampin-resistant M. tuberculosis was up to 60 days. It seems that microplate Alamar blue assay is very useful, fast, and cheap in determining the susceptibility of rifampin-resistant M. tuberculosis.

5.1. Conclusions

In general, it can be concluded that oxidation-reduction tracing is simple, reliable, and low cost. In developing countries, tuberculosis laboratories can use the Alamar blue test as a rapid and inexpensive method to determining susceptibility or resistance to M. tuberculosis. In this study, it was found that the Alamar blue colorimetric method can be used as a simple, reliable, and fast method to determine rifampin-resistant M. tuberculosis.

Acknowledgements

Footnotes

References

  • 1.

    Franzblau SG, Witzig RS, McLaughlin JC, Torres P, Madico G, Hernandez A, et al. Rapid, low-technology MIC determination with clinical Mycobacterium tuberculosis isolates by using the microplate Alamar Blue assay. Assay. J Clin Microbiol. 1998;36(2):362-6. doi: 10.1128/jcm.36.2.362-366.1998.

  • 2.

    Chauca JA, Palomino JC, Guerra H. Evaluation of the accuracy of the microplate Alamar Blue assay for rapid detection of MDR-TB in Peru. Int J Tuberc Lung Dis. 2007;11(7):820-2. [PubMed: 17609061].

  • 3.

    Coban AY, Uzun M, Akgunes A, Durupinar B, et al. Comparative evaluation of the microplate nitrate reductase assay and the rezasurin microtitre assay for the rapid detection of multidrug resistant Mycobacterium tuberculosis clinical isolates. Mem Inst. 2012;107(5):578-81. doi: 10.1590/s0074-02762012000500002.

  • 4.

    Migliori GB, Matteelli A, Cirillo D, Pai M. Diagnosis of multidrug-resistant tuberculosis and extensively drug-resistant tuberculosis: Current standards and challenges. Can J Infect Dis Med Microbiol. 2008;19(2):169-72. doi: 10.1155/2008/857901.

  • 5.

    Gomez-Flores R, Gupta S, Tamez-Guerra R, Mehta RT. Determination of MICs for Mycobacterium avium-M. intracellulare complex in liquid medium by a colorimetric method. J Clin Microbiol. 1995;33(7):1842-6. doi: 10.1128/jcm.33.7.1842-1846.1995.

  • 6.

    Yajko DM, Madej JJ, Lancaster MV, Sanders CA, Cawthon VL, Gee B, et al. Colorimetric method for determining MICs of antimicrobial agents for Mycobacterium tuberculosis. J Clin Microbiol. 1995;33(9):2324-7. doi: 10.1128/jcm.33.9.2324-2327.1995.

  • 7.

    O'Brien J, Wilson I, Orton T, Pognan F. Investigation of the Alamar Blue (resazurin) fluorescent dye for the assessment of mammalian cell cytotoxicity. Eur J Biochem. 2000;267(17):5421-6. doi: 10.1046/j.1432-1327.2000.01606.x.

  • 8.

    Jimma W, Ghazisaeedi M, Shahmoradi L, Abdurahman AA, Kalhori SRN, Nasehi M, et al. Prevalence of and risk factors for multidrug-resistant tuberculosis in Iran and its neighboring countries: systematic review and meta-analysis. Rev Soc Bras Med Trop. 2017;50(3):287-95. doi: 10.1590/0037-8682-0002-2017.

  • 9.

    Tarashi S, Fateh A, Jamnani FR, Siadat SD, Vaziri F. Prevalence of Beijing and Haarlem genotypes among multidrug-resistant Mycobacterium tuberculosis in Iran: Systematic review and meta-analysis. Tuberculosis. 2017;107:31-7. doi: 10.1016/j.tube.2017.03.005.

  • 10.

    Farnia P, Mohammadi F, Mirsaedi M, Zia Zarifi A, Tabatabee J, Bahadori M, et al. Bacteriological follow-up of pulmonary tuberculosis treatment: a study with a simple colorimetric assay. Microbes Infect. 2004;6(11):972-6. doi: 10.1016/j.micinf.2004.04.017.

  • 11.

    Caviedes L, Delgado J, Gilman RH. Tetrazolium microplate assay as a rapid and inexpensive colorimetric method for determination of antibiotic susceptibility of mycobacterium tuberculosis. J Clin Microbiol. 2002;40(5):1873-4. doi: 10.1128/jcm.40.5.1873-1874.2002.

  • 12.

    Viera AJ, Garrett JM. Understanding interobserver agreement: the kappa statistic. Fam Med. 2005;37(5):360-3. [PubMed: 15883903].

  • Copyright © 2021, Author(s). This is an open-access article distributed under the terms of the Creative Commons Attribution-NonCommercial 4.0 International License (http://creativecommons.org/licenses/by-nc/4.0/) which permits copy and redistribute the material just in noncommercial usages, provided the original work is properly cited.
    COMMENTS

    LEAVE A COMMENT HERE: