The best way to judge earnings call Q&A quality is a composite score built from signals such as semantic alignment between questions and answers, the information content of analyst questions, and vocal listenability. Research on semantic similarity links evasive answers to weaker stock returns, and Regulation FD (Form 8-K Item 2.02) defines what must become public. Our audit-traceable process uses public records.
TL;DR:
- Score only the Q&A segment, because scripted remarks can inflate similarity and listenability, obscuring the spontaneous answers the analysis aims to assess.
- Calibrate similarity thresholds with hand annotated examples, then compare each company with its own history and sector peers; raw scores have no standalone meaning.
- For small cap companies, require sufficient question and answer pairs before trusting averages, and give greater weight to qualitative review and disclosure status.
- When a call raises concerns, inspect flagged exchanges directly and check whether any new material information was promptly disclosed through an Item 2.02 filing.
Table of Contents
- What metrics capture Q&A quality and why they matter
- How to measure Q&A quality from transcripts and audio
- Interpreting scores and common patterns investors should watch
- Practical checklist: quick Q&A quality screening for analysts and investors
- How a forensic analytics workflow operationalizes Q&A quality
- Where Q&A quality metrics are headed
- Get reproducible Q&A quality scores and benchmarks
- FAQ
- Sources
What metrics capture Q&A quality and why they matter
Q&A quality is not a single property. It is the interaction of multiple measurable constructs, some of which have been tied to market behavior in published research.
- Semantic similarity: each analyst question and the executive answer are converted to embeddings, and cosine similarity measures how closely the answer addresses the question. Scores are aggregated to a call-level measure. Low similarity indicates an answer that drifts from the question asked.
- Analyst question quality: a machine-learning measure of question quality predicts information-rich management responses, larger post-call forecast revisions, and stronger intraday market reactions. Quality rises when macro uncertainty rises and when analysts have greater expertise.
- Vocal delivery and listenability: speech models score how comprehensible a manager's delivery is. Listenability research correlates it with stronger analyst interactions, higher price responsiveness, and tighter bid-ask spreads, and finds that delivery tends to deteriorate when managers present negative news.
- Conversational engagement: linguistic style matching between analysts and managers serves as a proxy for back-and-forth dialogue, and higher verbal coordination correlates with positive absolute stock returns.
The empirical payoff is the reason to measure at all. The semantic similarity study links aligned answers with stronger performance than evasive answers, and reports that low-similarity answers were often judged evasive, according to the semantic similarity study. That finding turns a qualitative impression ("they dodged that one") into a quantity an analyst can track across calls and cohorts.
Engagement itself carries information. Research on calls with no analyst questions found weaker liquidity and price responses, which suggests that thin analyst participation is a signal in its own right and not merely an absence of data.
How to measure Q&A quality from transcripts and audio
A reproducible measurement pipeline has several stages. Each can be run by an individual analyst with public inputs.
- Source the record and confirm disclosure status. Collect the post-call transcript, the audio or webcast, and the Form 8-K. SEC staff guidance states that material, previously undisclosed information revealed during an earnings call Q&A must be promptly made public, and that posting a transcript, audio file, or slides can satisfy Item 2.02. A call whose Q&A surfaced new material facts with no corresponding posting is itself a finding.
- Attribute speakers. Tag every turn as analyst, executive, or operator. Misattribution corrupts every downstream score.
- Pair questions with answers. Multipart questions and multi-executive answers need a consistent rule, such as splitting by sub-question or concatenating answers. Summarize long turns to standardize length so verbosity does not distort similarity.
- Compute text scores. Embed each pair, calculate cosine similarity, and aggregate to a call-level score using a fixed method (mean, median, or share below a threshold). The aggregation choice must be documented and held constant across the sample.
- Compute audio scores where audio exists. Deep-learning acoustic models measure vocal delivery quality and have been linked to analyst information production and price responsiveness. Apply them when you have clean audio for the Q&A segment specifically, not the prepared remarks.
- Validate. Hand-annotate a small sample of pairs, compare your labels with the machine scores, run rolling windows to detect drift, and test whether scores line up with post-call forecast revisions and intraday reactions. The FinArgQuality dataset defines quality dimensions for managers' arguments and offers an annotated baseline for this check.
Pro Tip: Score the Q&A segment on its own and keep the prepared remarks out of the sample. Scripted remarks are written to be clear, so including them inflates both similarity and listenability and hides the spontaneous behavior you are trying to observe.
Common pitfalls follow from the steps above. Transcript vendors differ in how they handle crosstalk and inaudible passages. Embedding models differ in their similarity scales, so thresholds calibrated on one model do not transfer to another. And a single call is a noisy observation: the measure becomes informative when you compare a firm against its own history and against its cohort.
Interpreting scores and common patterns investors should watch
Raw similarity values have no intrinsic meaning. They acquire meaning through calibration against a labeled sample and a peer distribution. The documented relationship is the anchor: the published results associate low-similarity answers with perceived evasion and aligned answers with stronger firm performance. A practical labeling scheme follows from that: treat the bottom tail of your calibrated distribution as "evasive-leaning," the middle as "partially responsive," and the upper tail as "aligned."
Patterns worth monitoring:
- Fewer questions or poorer listenability around negative surprises. Research shows vocal delivery often worsens when managers deliver negative or transitory news, so a sudden drop in delivery quality against the firm's own baseline deserves a closer read.
- Thin analyst engagement. Calls with no questions are associated with weaker liquidity and price responses, as the interaction study documents.
- High interaction quality under uncertainty. Manager-analyst interaction quality is positively associated with future firm performance and immediate positive market responses, especially when information uncertainty is high.
- Composite archetypes. Combine the three signals into buckets such as aligned and confident, aligned but flat in delivery, or evasive and uncertain. A single low reading may be transient; a persistent pattern across several quarters points to something structural.
Operational caveats limit how far any single score should be pushed. Firms with thin analyst coverage produce few pairs, so call-level scores have wide error. Industries differ in how technical and multi-part questions are, which shifts similarity baselines. And transcription quality varies, so a low score can reflect a bad transcript instead of an evasive executive. Benchmark within sector and cohort before drawing conclusions, and treat outliers against peers as the unit of analysis.
Practical checklist: quick Q&A quality screening for analysts and investors
Use this list to screen one call or to monitor a watchlist, in roughly this priority order.
- Record availability: transcript and audio both exist for the full Q&A. Without them, no further scoring is reliable.
- Unique analyst questions: count distinct questioners. A very small count lowers confidence in every other metric.
- Average question-to-answer similarity: compare against the firm's own trailing calls and its sector cohort.
- Share of evasive responses: the proportion of pairs falling below your calibrated threshold.
- Listenability flag: a marked drop in vocal delivery quality relative to the firm's baseline.
- Disclosure status: whether the transcript or webcast is posted and whether an Item 2.02 Form 8-K covers any new material information.
- Follow-up behavior: whether analysts return with follow-ups on the same topic, which often signals an unsatisfying first answer.
Weighting should reflect coverage. For large-cap names with many questioners, lean on similarity and evasive-share, since the sample is rich. For small-cap names, set a minimum pair count before trusting any similarity average, and give more weight to qualitative reading and disclosure status.
When a call flags low quality, take three steps: read the flagged pairs directly rather than relying on the score, check the filing record for any Item 2.02 posting, and prepare follow-up questions for management or investor relations. For disclosure-specific follow-ups, our posts on spotting selective disclosure patterns and Regulation FD implications cover the next steps in detail.
Teams preparing for investor meetings can also study how others frame engagement, such as this external perspective on tracking investor engagement, which treats question and slide interactions as measurable signals.
How a forensic analytics workflow operationalizes Q&A quality
We conduct audit-traceable analysis using public records. Our analysis is based on public records:
- Source collection: we analyze publicly available company disclosures.
- Analysis: we assess company disclosures.
- Scoring: we evaluate companies using a consistent methodology.
- Evidence-based evaluation: our analysis is based on public records.
- Benchmarks and archetypes: we compare companies and classify them into performance archetypes.
Our audit-traceable scoring makes the evidence behind results available for review. Our forensic audit framework and our piece on scores analysts can trace to quotes describe the evidence standard. Technical detail on data sources and scoring lives on our methodology walkthrough.
Where Q&A quality metrics are headed
Speech and language models will keep improving, and longitudinal benchmarks will make single-call scores far easier to place in context. I also expect regulators to look harder at patterns of selective disclosure across many calls instead of one episode at a time. Investors should prefer workflows that cite their evidence and validate against human-labeled samples, because a score you cannot trace to a quote is an opinion. The firms that hold up best under this scrutiny will be those whose answers match the questions asked.
— Glen
Get reproducible Q&A quality scores and benchmarks
Institutional readers can access our forensic analysis. Our offerings include:
- Per-company analysis: a forensic report evaluating a public company using public records.
- Cohort Pulse and Sector Sweep: peer benchmarking that places a call in its sector context, which is how outliers become visible.
- Methodology transparency: every score is documented and open to inspection.
Review our methodology, then request access to per-company analysis or see how cohort products are structured on our pricing page. Prices are available on request.
This article is general information, not a substitute for advice from a qualified financial advisor. Consult a qualified financial professional about your own circumstances before acting on anything here.
FAQ
What is a good quality of earnings ratio?
No single ratio defines good earnings quality. Assessments require a multidimensional, rolling evaluation across periods, because income-statement metrics alone are insufficient. For call quality specifically, compare a firm's scores against its own history and its sector cohort.
How to measure quality of earnings?
Compare periods, accounting policies, ratios, cash flows, and management commentary, as CFA Institute guidance describes. High-quality reporting is relevant and faithfully represents economic reality. Q&A analysis adds a check on whether management's spoken answers match the disclosures.
What is an earnings quality assessment?
It is a structured review of how well reported results reflect underlying economic performance. It looks across multiple dimensions and several periods instead of a single figure. Call Q&A scoring complements it by testing how directly executives address analyst questions.
How do investors judge earnings call Q&A quality?
Combine semantic alignment between questions and answers, analyst question quality, and vocal listenability, then benchmark against peers. Validate machine scores with a small hand-annotated sample, and confirm any new material information was posted under Form 8-K Item 2.02.