In this study, five prestigious international otolaryngology journals with high impact factors were reviewed for studies published in the subfield of laryngology. The journals reviewed were The Laryngoscope (n = 18 research), Auris Nasus Larynx (n = 15), American Journal of Otolaryngology (n = 16), Journal of Otolaryngology-Head and Neck Surgery (n = 17) and European Archives of Oto-Rhino-Laryngology (n = 14) and a total of 80 articles were evaluated. The purpose of selecting these journals was to represent internationally reputable journals with high methodological writing standards in the otolaryngology field.

Our inclusion criteria were that articles were available online, had a methods section, and included statistical analysis. These criteria ensured the article provided sufficient methodological information for a study-specific methodological assessment. Excluded articles were defined as articles that focused primarily on surgical techniques, clinical guidelines, expert opinion reviews, case series, and case reports. Such studies were excluded because they usually did not include a standardized methodology section or statistical analysis. A total of 60 original research articles that met the inclusion criteria were eligible for analysis.

The sixty articles were categorized into six research type categories according to their study design. These categories were: (1) Cell culture studies (in vitro; n = 10), (2) Animal experiments (in vivo; n = 10), (3) Prospective clinical studies (n = 10), (4) Retrospective clinical studies (n = 10), (5) Systematic reviews (n = 10), and (6) Artificial intelligence-based studies (n = 10). This categorization was made to evaluate the methodology design ability in the field of laryngology across different study types. Each category represents the types of studies commonly encountered in the laryngology literature.

From the 60 articles categorized into categories, five articles of each type were selected by simple random sampling to form the final analysis sample. An online random number generator tool was used for randomization (Random Number Generator; https://www.random.org). In this way, 30 articles (6 categories × 5 articles) were selected and included in the analysis. Selecting an equal number of studies from each category allowed for a balanced comparison of the performance of large language models in different types of research. The distribution of the final 30 included articles according to journals was The Laryngoscope (n = 9), European Archives of Oto-Rhino-Laryngology (n = 6), Auris Nasus Larynx (n = 5), American Journal of Otolaryngology (n = 5) and Journal of Otolaryngology-Head and Neck Surgery (n = 5) (See Table 1).

To simulate realistic research scenarios, the first author (NT) systematically reviewed the methodology sections of the selected original research articles and prepared concise summaries for each. These summaries served as the foundation for generating study-specific prompts, designed to reflect plausible and clinically relevant research inquiries. In line with this structure, three core questions were deliberately posed to the AI models, each corresponding to a fundamental element of research design: (1) sample size and patient identification, (2) data collection and measurement-evaluation methods, and (3) statistical analysis. This triad was informed by established methodological frameworks such as the FINER criteria (Feasible, Interesting, Novel, Ethical, Relevant) and the PICOT structure (Population, Intervention, Comparison, Outcome, Time), which emphasize the importance of these components in the formulation of scientifically rigorous and clinically meaningful studies [11]. Moreover, this approach aligns with widely accepted principles in protocol development and meta-research methodology [12, 13]. By structuring the AI interaction in this targeted manner, the study aimed to assess not only the factual integrity of the generated outputs but also the extent to which the models could emulate expert-level reasoning in methodological design.

The study evaluated the ability to build methodologies using two different LLMs: ChatGPT-4 (OpenAI, USA) and Gemini (Google, USA). To test the LLMs’ ability to build a research methodology, each LLM was given a standardized prompt followed by a research methodology questions. The prompt preceding each question is shown below: You are an AI with world-class academic expertise in laryngology. You also have expertise in building research methodology and understand the best and current practices and strategies in research methodology. When you are given questions about building research methodology, take a deep breath and think step-by-step to create an accurate and up-to-date methodology design. When given a question, answer; do not explain your reasoning. This prompt is written based on published work describing techniques to improve LLM accuracy through prompting balanced with instructions to avoid overly detailed answers to improve high-throughput analysis [14]. For example, LLM performance tends to improve when the prompt assigns a relevant role (such as an expert in the relevant field), uses phrases such as “take a deep breath” and “think step by step” to encourage step-by-step planning, and provides explicit instructions. The aim is to ensure that each model constitutes the Method part of a scientific investigation under the same conditions.

As an illustrative example, Table 2 presents the study-specific prompt along with the comparative methodology outputs generated by ChatGPT-4 and Gemini 1.5 Flash for a retrospective investigation examining the use of basic fibroblast growth factor (bFGF) in the treatment of age-related vocal fold atrophy [15].

The ChatGPT-4 models were accessed through the OpenAI interface and the Gemini model through the respective platform. The responses from the models were recorded verbatim without any correction or intervention. Each LLM output was saved as a separate Microsoft Word document to be used for later evaluation and analysis (see Apendix). The study did not require ethics committee approval (IRB) as no human participants or living material were used in the study.

Two independent expert reviewers (E.T. and E.B.) evaluated the quality of the methodology texts produced by the large language models. Both reviewers are associate professors specialized in otolaryngology, with subspecialty expertise in laryngology. Each has over a decade of clinical and academic experience in the field. They independently scored each AI-generated methodology response using a 7-point Likert scale, following previously validated evaluation approaches in the literature [16,17,18]. At the time of the assessment, the assessors did not know which model the texts came from; all responses were randomly coded and presented. The Likert scale was defined with a score of 1 representing the poorest methodological quality and a score of 7 representing excellent methodological quality. Accordingly, Likert 1: very poor (range 1-1.85), Likert 2: relatively poor (range 1.86–2.71), Likert 3: poor (range 2.72–3.56), Likert 4: neither good nor poor (range 3.57–4.43), Likert 5: good (range 4.44–5.28), Likert 6: quite good (range 5.29–6.13), Likert 7: very good (range 6.14-7). Each full score was scaled to correspond to equal intervals of approximately 0.85 points between 1 and 7. In this way, the scale was adjusted to reflect gradual differences in methodology quality accurately. The scores given by both raters were recorded in Microsoft Excel (Microsoft Corp., Redmond, WA).

Descriptive statistics were calculated using the mean scores of the two raters. Weighted kappa was used to evaluate the agreement between two raters for ordinal data. To ensure reliability, bootstrap resampling with 1000 iterations was used to compute 95% confidence intervals for kappa values. The Weighted Agreement Score (WAS) calculates the proportion of ratings when the difference between two raters is within a set tolerance, such as a one-category difference. This score offers a simple way to assess overall agreement, especially for ordinal data. Because of the significant agreement observed, the average scores of the two raters were determined and used to compare the two language models. The Shapiro-Wilk and Kolmogorov-Smirnov tests were employed to assess the normality of the data distribution. Cohen’s d method was used to calculate effect sizes, quantifying the extent of the difference. An independent samples t-test was utilized to examine the performance of the two language models for normally distributed data. We employed one-way ANOVA for normally distributed data and the Kruskal-Wallis test for non-normally distributed data to assess the variations in response accuracy among article types for ChatGPT and Gemini. All analyses and visualizations were conducted via R Studio.A p-value of < 0.005 was considered statistically significant.