What happens to the patients who are still alive?

A patient enrols in a cancer study. Years pass. The study closes on a fixed date, the same date for everyone, and it does not matter where each patient’s disease happens to be on that day. Some patients have had their event by then: a recurrence, confirmed and dated. Others have not, and are alive and disease-free when the clock stops. For them the true answer is not yet.

Every chapter so far has asked a yes-or-no question: resistant or not, bound or not, named the same or not. This chapter asks a harder kind of question. It asks when something happens. The data is cancer survival, with real patients, real follow-up time and real recurrence, and it brings a new way for the label itself to lie.

Suppose the patients still alive are treated as a no, the same as someone followed for ten years who never recurred. Then the model has been told something false. A patient followed for eight months with no event yet and a patient followed for eight years with no event ever look identical in that label. Those are not the same fact. One is early. One is settled.

Censoring is not unique to cancer data. Any study with a fixed end date and an event that might happen later has it: component failure, customer churn, reoffending after release. Biology just makes the stakes obvious.

There is a common wrong move here, and I named it before I could be tempted by it. Drop everyone still alive at the end, and keep only the patients with a known event. That feels safe. In fact it deletes exactly the patients who are doing best, because good outcomes take longer to prove than bad ones. The remaining data is quietly enriched for recurrence, and the model learns a population that does not exist.

So the label for this chapter is two numbers together: how long a patient was followed, and whether the thing being watched for happened before the watching stopped. The patients still alive at the end are not missing data, and they are not a problem to clean up before modelling starts. They are the central fact of the chapter. Model around them carelessly and every number that follows is wrong.

4.1 Why gradient boosting, and why it needs a different objective

There is a reason Chapter 3 ended on random forests. Averaging many trees is one way to reduce error. Building trees one at a time, each one correcting what the last one got wrong, is another. That second way is gradient boosting, the next model on the list set out in Chapter 2. It had already made a quiet appearance in Chapter 3, as a comparison.

Every model before this one was trained to predict one number and checked against one answer. This one predicts a number and is checked against two things at once: whether the event happened, and how long it took. Ordinary gradient boosting cannot do that. The measure a model is trained to improve is called its objective function, and it sets what counts as a good prediction. Without an objective function that understands censoring, gradient boosting does exactly what I warned against above, and scores a patient still doing well at eight months the same as one confirmed disease-free at eight years.

The fix is called the Cox partial likelihood, and the idea behind it is older than gradient boosting itself. Do not ask the model to predict a survival time directly. Ask it for a risk score, and judge that score by how well it puts patients in the order in which their events came.

In practice it works like this (Figure 4.1). At the moment each event occurs, look at everyone still at risk, meaning everyone who has not yet had the event and is still being followed. The patient it happened to should have scored higher risk than everyone else still in the running at that moment. Do that at every event in the data and add it up, and the result is a single number the model can be trained to improve. A censored patient takes part in every one of these comparisons up to the last date anyone checked on them, and in none after it. The model is never asked to know exactly when. It is asked to know who is more at risk than whom, at every moment someone’s risk became real.

The two numbers of the label reach the model as one. Listing 4.1 shows how: the follow-up time is kept positive for a patient who had an event and made negative for one who was censored.

def cox_labels(time, event):
    """XGBoost survival:cox labels: positive time for events, negative for censored."""
    time = np.asarray(time, dtype=float)
    event = np.asarray(event).astype(bool)
    return np.where(event, time, -time)

One consequence was worth sitting with before any result existed. A model can have excellent concordance, putting patients in the right order of risk, and still be useless for telling one specific patient how long they have. Ranking who is at higher risk is a different skill from predicting a date. This chapter is about the first skill, and I named that limit before a number could make it inconvenient to say.

4.2 The rules

I wrote the rules down before I had loaded a single row.

The label is disease-free survival, not overall survival and not progression-free survival on treatment. Death from any cause is in there deliberately. A patient who dies in a car accident before their cancer could recur is a hard case, and different studies handle it differently. I counted it as an event, not a censor. Pretending that patient’s follow-up simply continued would be its own quiet lie. I recorded the choice in advance so that I could not change it later if the number looked better the other way.

Starting the clock at diagnosis was a choice too. Enrolment and the end of primary treatment would each give a different clock, and three different clocks make three different chapters.

Listing 4.2 is the event rule as the pipeline applies it to each patient. The earlier of a dated recurrence and a dated death becomes the event. A patient with neither is censored at the latest recorded contact.

candidates = [d for d in (first_recurrence, first_death) if d is not None]
if candidates:
    time, event = min(candidates), 1
    if first_recurrence is not None and first_recurrence == time:
        event_type = "recurrence"
    else:
        event_type = "death"
else:
    contacts = [c for c in ([number(row.get("last_contact_days_to"))]
                            + fu_contact[barcode] + fu_death[barcode])
                if c is not None]
    if not contacts:
        audit["dropped_no_time"] += 1
        continue
    time, event, event_type = max(contacts), 0, "censored"

The thirty-day threshold exists because so short a follow-up says nothing, and including those patients would let the model learn from noise dressed up as signal. The grouped split did not need deciding, only restating.

The concordance index asks a narrower question than accuracy does. The model learns to get orderings right, and it is scored the same way.

The baseline is stage alone: whatever a clinician already knows from TNM staging, and nothing else. TNM records how far the primary tumour has grown (T), whether the cancer has reached nearby lymph nodes (N) and whether it has spread to distant parts of the body (M), and the three are combined into a single stage. If gradient boosting could not beat stage, the honest chapter would be the one that said so.

4.3 Four ways this could be wrong

I did not know whether this chapter would work, so I wrote down why it might not before I found out (Figure 4.2).

The first is the cohort. Public cancer cohorts are not a random sample of cancer patients. They are the patients whose data made it into a study, at institutions with the funding to run one, filtered by inclusion criteria that already excluded the sickest and most complicated cases. Whatever this model learns, it learned from that population and not from cancer patients generally.

The second is structural rather than a matter of sampling. Stage, grade and tumour size are correlated with each other. Bigger tumours tend to be later stage, and later stage tends to mean higher grade. A model can look as if it found three independent signals when it found one signal, badly, three times. Chapter 3 met a form of this, which I call circularity: a model detecting the tool that made the label rather than the biology underneath it. The version here is one clinical fact, detected through three correlated proxies and reported as three important features instead of one.

This data turned out to record neither grade nor tumour size, so the check for this warning ran on stage and the other descriptors that were recorded, with pT, the T category of TNM, standing in for tumour size, and pN, the N category, for the lymph nodes. Stage is built partly from those two, so the check still asked the same question.

The third is a small event count. Survival models are limited by how many events occurred, not by how many patients were enrolled. A cohort of a thousand patients with sixty events behaves, for fitting purposes, much closer to a cohort of sixty. If confirmed recurrences in this cohort number in the low hundreds, the honest ceiling is lower than the sample size suggests.

The fourth is the one I was most likely to get away with unless I checked for it directly. How closely a patient is followed can itself carry information that has nothing to do with biology. Some patients are followed more closely because a clinician was already worried. That worry, not the tumour, can end up predicting outcome.

None of these breaks the chapter. Each one changes what a result would mean. Each one would be checked and reported, whether or not it changed the headline number.

4.4 What happened

The cohort is 586 patients from two public colorectal cancer datasets. Of these, 185 had an event: the cancer came back, or the patient died. The other 401 had no event with a date, as of the last time anyone checked on them. One of them did have a recurrence on record, but without a date it could not start a clock, so that patient was censored with the others.

The model was given ten pieces of information about each patient, things like cancer stage, lymph node status and treatment history.

I tested it on 176 patients the model had never seen, 56 of whom had an event. The model scored 0.7289 on the concordance index. It got the ordering of a pair of patients right about seventy-three times out of a hundred. To score stage alone on the same patients, I fitted a Cox proportional hazards model, a regression model built for survival data, to stage and nothing else on the 410 training patients, giving patients whose stage was not recorded a level of their own. Stage alone scored 0.6743.

Besides the four warnings, I had committed to two checks before I knew what they would show.

The first asked what happens when only colon cancer is considered, leaving out rectal cancer. I refitted both models on the 297 colon patients in the training set, 96 of whom had an event, and scored them on the 128 colon patients held out, 42 of whom had one. The model scored 0.7228 against 0.706 for stage. That is still an improvement, but a smaller one, and this time the gap is not clearly bigger than what chance alone could produce. Both groups agree that the model does better. Only the larger, mixed group is confident enough to say that the improvement is real rather than noise.

The second asked whether knowing how many mutations a tumour carries adds anything. Of the 586 patients, 540 had this data. Their mutation counts split into two groups, as expected in this cancer: 77 of the 540 tumours, 14.3%, carry more than 10 mutations per megabase, which is the hypermutated fraction already known in colorectal cancer. So I trust the measurement. I refitted the model with and without mutation count on the patients who had it, and scored both on the 159 of them held out, 52 with an event. Without mutation count the model scored 0.7243, and with it 0.7302. The difference, 0.0059, is less than a hundredth, and its 95% interval runs from negative 0.0055 to 0.0186. That takes in zero, so the change could easily be nothing at all. There is no evidence here that mutation count helped this model predict risk.

Then the four warnings. The first, that this cohort is not a random sample of cancer patients, cannot be tested from within the cohort. Every number in this chapter describes these patients, not colorectal cancer patients in general.

The second warning asked whether the model would find one real signal and report it as two or three separate important features, making it look as if it used more information than it did. That one did happen. The two most important features were stage and pN, the lymph node category, and the two move together. Across the 566 patients with both recorded, their rank correlation is 0.8289, where 1 would mean that they always rise together. They are largely one fact about the patient, counted twice.

The third was the event count. Before fitting, I had fixed a floor of 50 events, below which a result would carry a warning. The main split, colon and rectal cases together, cleared it, with 129 events among the training patients and 56 among those held out. The colon-only test set did not. It had 42, and at that number of events the colon gap cannot be told apart from chance.

The fourth asked whether being checked on more often, because a doctor was more concerned, itself went with a worse outcome, so that the model could have learned the doctor’s worry rather than anything about the cancer. I found no evidence that it did. The test I had fixed before looking at the data counted each patient’s follow-up contacts in the first 90 days after diagnosis, and asked whether that count predicted an event after those 90 days. It could not run. Follow-up forms in these datasets are filed once a year, so only 4 of the 544 patients still at risk at 90 days had any contact inside the window. After seeing that, and before testing anything against outcome, I widened the window to the first 365 days. Each extra contact in that year went with a hazard ratio of 0.455 for an event afterwards. A hazard ratio of 1 would mean no effect at all, and the 95% interval around 0.455, from 0.1678 to 1.2341, takes in 1.

4.5 What the result does and does not show

The model ranks patients by their risk of recurrence or death somewhat better than stage alone does. That is the finding.

The improvement over stage holds up only in the larger, mixed group. On colon cases alone it points the same way but cannot be told apart from noise. The second warning came true, so the model’s list of important features overstates how many separate signals it used. And the result says nothing about how long any particular patient has.

Four chapters have now used four models. Logistic regression drew a straight line through a decision. A single decision tree could be read out loud, one split at a time. A random forest let many trees vote together, each one blind to what the others saw. Gradient boosting built trees one after another, each correcting what the last got wrong, and here it was adapted to patients whose outcome had not happened yet. That is the standard path through tree-based models, the one most textbooks follow in roughly the same order, and it ends here. Chapter 5 sets out to use a support vector machine, the next step on that same standard progression.

This chapter is rewritten from the posts for Days 40 to 46 of the series.