Gut bacteria and cardiovascular risk

Somewhere in your gut, bacteria are doing something no cell in your body can do.

Your own enzymes cannot break the bond between carbon and nitrogen in a choline molecule. Certain gut bacteria carry one that can, a glycyl radical enzyme called TMA-lyase, and they use it constantly. They cleave the choline and carnitine that come from the steak or eggs you ate, and release a waste gas called trimethylamine, or TMA. The gas crosses into your bloodstream and reaches your liver, where a single enzyme, FMO3, oxidises it into trimethylamine N-oxide. TMAO.

From there it does real damage. It pushes macrophages to swallow more cholesterol than they should, turning them into the foam cells that build atherosclerotic plaque. It blocks the pathway that clears cholesterol back out of your arteries. It makes platelets clot more readily than they should. Elevated TMAO is now one of the more consistently replicated predictors of cardiovascular risk in the literature, and it predicts risk independently of cholesterol itself.

That gives this chapter its question. The enzyme is bacterial, and the bacteria that carry it can be identified. If a support vector machine is shown only the abundance of those TMA-producing bacteria, with no host genetics and no cholesterol panel, can it predict cardiovascular risk from that alone?

Measuring TMAO itself needs a specialised laboratory assay, and almost nobody who could benefit from knowing their risk will ever have it run. The question underneath the question is whether the presence of the machinery that makes TMAO can stand in for measuring TMAO at all.

One limitation, named now rather than discovered later. TMA-lyase only matters if there is substrate to cut. A person can carry every one of these bacteria and still produce almost no TMAO, if choline and carnitine barely appear in what they eat. Without diet data, a model of this kind has a hard ceiling on what it can know, and that ceiling is part of the chapter, not a footnote added after the fact.

5.1 The enzyme is not the organism

The plan as I first stated it was to give the model the abundance of bacteria known to carry TMA-lyase. That sentence hides a problem, and the problem decided what data the chapter would need.

The gene for choline TMA-lyase is called cutC, and it does not follow the family tree. Two genera, Desulfosporosinus and Proteus, carry it in every sequenced genome. Almost nothing else does. Ninety-eight of 2,719 sequenced Escherichia coli genomes carry a cutC homologue, four per cent. The other ninety-six per cent do not. Same genus, same name in a results table, and only one in twenty-five can perform the reaction.

The gene spread by horizontal transfer, passed sideways between organisms rather than inherited down a lineage. It turns up in patches across four bacterial phyla, and it is absent from most members of most of the genera that carry it. So the obvious approach does not work.

Most public microbiome data is 16S sequencing. It reads one marker gene and reports which genera are present, and in what proportion. It does not report which genes those organisms carry. If I hand a model the abundance of a genus that sometimes carries cutC, I am not handing it the pathway, the genes that turn choline and carnitine into TMA. I am handing it a proxy for a proxy, and how good that proxy is depends on what fraction of the genus carries the gene in that particular person’s gut. 16S sequencing cannot measure that fraction.

Shotgun metagenomics can. It sequences everything, so cutC can be counted directly as a gene, whichever organism happens to carry it (Figure 5.1). The two methods are different experiments with different ceilings, and choosing between them was the first decision the chapter had to make in public.

5.2 Why a support vector machine was planned

The metagenomic design was planned around a support vector machine. Its features, the measurements the model would be given for each person, were a short list: the six genes of the pathway that makes TMA, each with a mechanistic reason to be there. Six numbers per person, a few hundred people, and a boundary that might curve rather than split cleanly. That is close to the exact problem support vector machines were built to solve.

The models of Chapters 2 to 4 all worked by cutting the data into groups. Is this gene above a certain level, yes or no? Then the model asks again. Stack enough of those cuts together and you get a decision tree, or a forest of them, or trees that correct each other. All of it is still cutting.

A support vector machine does something different. It measures the gap. Picture each sample as a point, placed according to its six gene abundances. The model looks at the healthy samples and the sick samples, finds the widest possible empty space between them, and draws its boundary straight down the middle of that space (Figure 5.2).

The width is the point. Most models are content with any boundary that separates the training data correctly. There are usually many such boundaries, and on the data already in hand they all look equally good. This model picks the one with the most room on either side, because room is what helps when the next patient does not look exactly like anyone it has seen before.

Something else follows from this. Only the patients sitting closest to the boundary shape it. Everyone comfortably inside their own group could be removed from the data, and the boundary would not move. On a small group of patients, that is useful, because the model is defined by its hardest, most borderline cases and not by the easy majority. Those borderline patients are called support vectors, a vector being one patient’s list of measurements, and they give the model its name.

Sometimes no straight line can separate two groups, however it is angled. For that case the model can change how it measures distance between patients, a rule called the kernel. Changing that rule amounts to stretching the space until a straight line does work. Nothing new is measured. Only the notion of near and far changes, and a straight boundary in the stretched space comes back curved when it is drawn on the original data.

Whether the straight version or the curved version would work better here was something to test, not assume, and it was to be decided using only the training data, before any final test.

5.3 The rules

The rules for the metagenomic design were written before anything was downloaded. The last three items in the box belong to the narrower dataset the chapter later moved to, described below.

The first rule follows from the way cutC is spread between bacteria. Genus abundance is a broken proxy for the pathway, and with it I would be measuring the wrong thing carefully. Normalising to total reads, the number of DNA fragments sequenced from the sample, stops a more deeply sequenced sample from looking like a sicker one. The clinical variables stay out because the question is what the pathway alone can carry; adding a clinical panel answers a different and much easier question.

The split is by person because some cohorts collect several stool samples from the same individual. If those samples were divided between training and testing, the model would meet a person in training, recognise them again in the test set, and score well without having learned anything about the pathway.

The confidence interval is there because a number built from a few hundred people has a range around it, and hiding that range would flatter the result.

The baseline is deliberately unglamorous. If six separate gene abundances cannot beat their own sum, the useful signal is simply how much pathway is present. Which genes, and in what proportion, would be adding nothing. That would be a real result rather than a failed chapter, and it would have been the headline.

5.4 Four ways this could be wrong

I wrote down four ways the chapter could be wrong before I knew which of them was true.

The first is that gene presence is not gene expression. A metagenome shows that cutC is there. It does not show that the organism is transcribing it, that the enzyme is working, or that TMA is being produced on the day the sample was taken. Every result of this design would be about capacity, not activity. Those are not the same claim.

The second is the substrate problem from the opening, still unsolved. Without dietary data, the model has a ceiling it cannot see past. If the cohort had diet records, I would report results with and without them. If it did not, that limitation would stay in the headline.

The third is that metagenomic abundances are compositional. They are proportions and sum to a fixed total, so one gene rising forces every other gene to fall, whether or not anything real changed. Correlations in such data can be produced entirely by that constraint. I committed to reporting results on a centred log-ratio transform as well as on raw proportions, and to treating any disagreement between the two as the finding. The transform takes the logarithm of each gene’s abundance and subtracts the average logarithm across all the genes in the same sample, so that each gene is read against the others in its own sample rather than as a share of a fixed total.

The fourth is geography. Gut microbiome composition varies enormously by population, diet and country, and a model trained on one cohort may be detecting where the samples came from rather than who has disease. If more than one cohort was available, I would test across them. If only one was, the result would travel no further than that population.

All four were written for the metagenomic design, and that design never ran, because the public data could not support it. The tests that did run, on the narrower dataset described next, used substances measured in plasma rather than genes counted in stool, so the first three do not apply to them. The fourth applies only in its fallback form. That dataset is a single cohort, and what it shows travels no further than that population.

5.5 A narrower dataset

The question had become whether TMAO helps predict cardiovascular risk on top of everything doctors already check. The data to answer that directly is not public. The studies that hold it keep it behind institutional approval, which can take months with no guarantee of a yes, and I decided not to chase it any further. What I found instead was something smaller.

It is a public dataset of 750 patients, all of whom had recently had a heart procedure, followed for nine months to see who had chest pain again. It measured TMAO and the three things TMAO is made from, choline, betaine and carnitine, in every single patient, with no gaps (Figure 5.3). The dataset lists a panel of 600 substances, those four among them, and 405 of the 600 were measured in every patient.

This dataset cannot answer that question. It has no medical history and no other risk factors, so there is nothing for TMAO to add value on top of. The patients had already had heart problems, so it says nothing about predicting a first heart attack in a healthy person.

What it can answer is narrower. Does TMAO carry any extra information beyond the three ingredients it is built from? And, separately, does a model built on all 600 substances hold up when it is tested properly, on patients it was not trained on, or does it only look good on paper?

No support vector machine ran on this dataset. I answered both questions with logistic regression, the model of Chapter 1. For the first, a logistic regression on the three ingredients was set against the same model with TMAO added, with no penalty on either. For the second, a logistic regression was given all 600 substances and penalised to keep its weights small, and the strength of that penalty was chosen inside each training set, so that the patients held back for scoring played no part in choosing it. Every model was scored on patients it had not been trained on, by cross-validation as in Chapter 2. The 750 patients were divided into five parts, each model was trained on four parts and scored on the fifth, in turn, and the whole division was made five times, with different parts each time.

I wrote down the rules for judging both answers before any model was fitted, but not before I had looked at the data. To check that the dataset was usable at all, I had already summarised TMAO and its three ingredients in each group of patients and run a rank-sum test on TMAO, so I knew which way the TMAO difference ran before the rules existed. That is a departure from an ideal pre-registration.

The rule for the first question, set out in the box earlier in the chapter, has two parts, and they test different things. An interval on the difference in area that excludes zero shows that the difference can be detected. The decision curve asks whether using the model with TMAO would lead to better decisions than using the model without it. Had the first part held and the second failed, the rise would have been reported as detectable but clinically negligible.

5.6 What came back

Two results came back.

The first answers the question about TMAO itself. Adding TMAO to a model of its own three building blocks, choline, betaine and carnitine, raised the area under the ROC curve for predicting who would have chest pain again from 0.848 to 0.873. More precisely, the model without TMAO scored 0.8475, with a confidence interval of 0.8132 to 0.8818, and the model with it scored 0.8731, with an interval of 0.8402 to 0.906. The difference was 0.0256, with an interval of 0.0075 to 0.0436, clear of zero. The decision curve of the model with TMAO lay above that of the model without it across 0.9821 of the range of thresholds, almost all of it. Both parts of the rule were met, so by the standard fixed before any model was fitted, that is a real improvement.

The second is the one I trust more. The model built from the whole panel, in effect the 405 substances measured in every patient, since the other 195 were empty for everyone, came back nearly perfect at separating the patients who had chest pain again from those who did not. Scored on patients it had not been trained on, its area under the ROC curve was 1.0, and its accuracy, the share of patients it put in the right group, was 0.9973. Even so, it only partly met the rule I had fixed for it in advance. Its accuracy, overall and within each group of patients, was far above the floor of 0.85 that the rule set, but its calibration slope, 2.97, lay outside the band of 0.80 to 1.25 that the rule also required. A slope that far from 1 means the predicted risks cannot be read as probabilities, even though the model ranked the patients almost perfectly.

A score that high on a hard clinical question is usually a sign to look closer rather than to celebrate. I looked closer. A large share of the substances in the panel, and not only TMAO, could each separate the two groups of patients on their own.

That changes how the first result reads. Ranked against the other 404 substances measured in every patient, TMAO comes in around the 64th percentile, behind roughly 144 others. One of those is betaine, one of its own ingredients and already part of the model that TMAO was added to. Listing 5.1 shows how that ranking is made: each substance is scored on its own by its area under the ROC curve, in whichever direction separates the two groups better, and TMAO is placed among them.

    outcome = frame[OUTCOME].to_numpy(int)
    complete = [m for m in metabolite_columns(frame) if frame[m].notna().all()]
    auc = pd.Series(
        {
            m: max(roc_auc_score(outcome, frame[m]), 1 - roc_auc_score(outcome, frame[m]))
            for m in complete
        }
    )
    precursor_auc = auc[list(PRECURSORS)]
    best = precursor_auc.idxmax()

    return PanelContext(
        marker=TMAO,
        univariate_auc=float(auc[TMAO]),
        percentile=100 * float((auc < auc[TMAO]).mean()),
        ranking_higher=int((auc > auc[TMAO]).sum()),
        panel_size=len(complete),
        best_precursor=str(best),
        best_precursor_auc=float(precursor_auc.max()),
        panel_median_auc=float(auc.median()),
    )

5.7 What the result does and does not show

These are results about a dataset more than about TMAO.

This is a public dataset, and like many public deposits it does not include, alongside the results, the order in which the lab processed the samples. Without that, there is no way to fully rule out that something about how the two groups were handled, rather than their biology, is doing some of the work. Instruments drift over a long run, for example, so if most samples from one group had been processed before most from the other, the drift alone could make many substances differ between the groups.

The deposit’s own numbering makes that more than a hypothetical. Samples 1 to 210 are exactly the 210 patients who had chest pain again, and samples 211 to 750 exactly the 540 who did not. The plan that held the rules also held a check on this. Within each group separately, so that the difference between the groups played no part, it asked whether each substance’s level rose or fell with the sample number. If the numbering had nothing to do with the measurements, about 5% of the 405 substances would appear to do so by chance. Among the patients who had chest pain again, it was 16.8%, and among the others 15.6%. That does not show the samples were run in numbered order, because the numbering need not be the order the lab used. It does mean that every difference between the two groups in this dataset may be tangled up with that order, to a degree the deposit cannot measure. That is a limit of what this particular dataset can tell us.

Seen against that, the TMAO number needs a caveat. I report it exactly as I said in advance I would. It cannot be read as strong evidence about TMAO specifically, given what the dataset can and cannot rule out.

The question of whether TMAO helps predict heart disease risk in people who are still healthy stays open. The code, the rules written before each test, and every result in this chapter are in the repository.

That makes five chapters and five models, though the fifth was planned and never run. Every chapter so far has paired one model with one biological question chosen to make that model earn its place. Here the support vector machine, which looks for the widest possible gap between two outcomes rather than any line that happens to separate them, was paired with the metagenomic question, and both results came instead from logistic regression, the model of Chapter 1. Chapter 6 takes a question where no single line or split will do, and a model that learns for itself which positions in a peptide act together.

This chapter is rewritten from the posts for Days 47 to 54 of the series.