Machine Failure Risk Prediction

This project builds an end-to-end supervised-learning workflow for machine-failure classification using the UCI AI4I 2020 Predictive Maintenance Dataset.

The notebook is the narrative analysis layer of the repository. Reusable implementation lives in the predictive_maintenance package:

• data.py — loading, schema extraction, and train/test splitting
• data_quality.py — validation checks
• eda_visualization.py — exploratory plots
• processing.py — feature engineering and preprocessing
• models.py — candidate models and hyperparameter searches
• evaluation.py — cross-validation, threshold selection, calibration, cost analysis, and final metrics
• interpretability.py — post-hoc explanations and error analysis
• plotting.py — evaluation and interpretation figures
• reporting.py — CSV reports, confidence-interval tables, summaries, and model artifacts

1. Problem Definition

Objective

Predict whether a machine will fail from a snapshot of its operating conditions:

• air temperature
• process temperature
• rotational speed
• torque
• tool wear
• product type

The task is strongly imbalanced, so the workflow emphasizes Average Precision, recall, F2-score, threshold selection and operational error costs rather than accuracy alone.

Operational meaning of errors

A false negative is a missed failure. It can represent unplanned downtime, production interruption or equipment damage.

A false positive is an unnecessary maintenance alert. It consumes inspection time but is generally less costly than an undetected failure.

The final operating threshold is therefore selected from training-set out-of-fold predictions by maximizing precision while satisfying a minimum recall requirement.

Synthetic-data limitation

AI4I 2020 is a synthetic benchmark generated from documented operating rules and stochastic processes. It is useful for demonstrating a rigorous ML workflow but its absolute performance must not be interpreted as evidence of production readiness.

Any real deployment would require temporal validation, machine-specific data, calibrated operating costs and monitoring for distribution shift.

RANDOM_STATE = 42
TEST_SIZE = 0.2
N_SPLITS = 5
MIN_RECALL = 0.8

2. Data Loading and Schema

Dataset shape: (10000, 12)

	name	role	type	demographic	description	units	missing_values
0	UID	ID	Integer	None	None	None	no
1	Product ID	ID	Categorical	None	None	None	no
2	Type	Feature	Categorical	None	None	None	no
3	Air temperature	Feature	Continuous	None	None	K	no
4	Process temperature	Feature	Continuous	None	None	K	no
5	Rotational speed	Feature	Integer	None	None	rpm	no
6	Torque	Feature	Continuous	None	None	Nm	no
7	Tool wear	Feature	Integer	None	None	min	no
8	Machine failure	Target	Integer	None	None	None	no
9	TWF	Target	Integer	None	None	None	no
10	HDF	Target	Integer	None	None	None	no
11	PWF	Target	Integer	None	None	None	no
12	OSF	Target	Integer	None	None	None	no
13	RNF	Target	Integer	None	None	None	no

Feature and target policy

UDI and Product ID are identifiers rather than operational predictors and are excluded from modeling.

TWF, HDF, PWF, OSF, and RNF are failure-mode flags used to construct the binary Machine failure target. They are never used as model inputs. They are retained only for post-hoc failure-mode analysis after the model and threshold are frozen.

Model-input features: ['Type', 'Air temperature', 'Process temperature', 'Rotational speed', 'Torque', 'Tool wear']
Target columns in source data: ['Machine failure', 'TWF', 'HDF', 'PWF', 'OSF', 'RNF']
Feature matrix shape: (10000, 6)
Target shape: (10000,)

	Type	Air temperature	Process temperature	Rotational speed	Torque	Tool wear
0	M	298.1	308.6	1551	42.8	0
1	L	298.2	308.7	1408	46.3	3
2	L	298.1	308.5	1498	49.4	5
3	L	298.2	308.6	1433	39.5	7
4	L	298.2	308.7	1408	40.0	9

Target construction

Machine failure equals 1 when at least one documented failure mechanism is active.

The mechanisms are separate but not necessarily statistically independent:

• TWF: stochastic tool-wear failure within a specified wear interval
• HDF: heat-dissipation failure based on temperature difference and rotational speed
• PWF: power failure based on mechanical power
• OSF: overstrain failure based on torque, tool wear, and product type
• RNF: random failure not recoverable from the operating predictors

HDF, PWF, and OSF are strongly recoverable from the available inputs because their rules use those inputs directly. TWF is partially predictable from tool wear, while RNF is irreducible from the provided features.

Rows consistent with failure-mode OR rule: 99.73%

	Row count
0	9652
1	324
2	23
3	1

The consistency rate should be at (or extremely close to) 100% — Machine failure is defined as the logical OR of the five failure-mode flags, so any deviation would mean either a labeling anomaly in this data pull or a misreading of the construction rule and should be resolved before trusting anything downstream.

failure_overlap counts how many of the five mechanisms are simultaneously active per row. Rows with more than one active mechanism matter for the Failure-Mode Detection analysis in Section 12: a false negative on such a row can't be attributed to a single failure mode, since more than one documented cause was present.

3. Data Quality

	na_count	null_count
Type	0	0
Air temperature	0	0
Process temperature	0	0
Rotational speed	0	0
Torque	0	0
Tool wear	0	0
Machine failure	0	0
TWF	0	0
HDF	0	0
PWF	0	0
OSF	0	0
RNF	0	0

	duplicate_feature_rows	duplicate_full_rows
0	0	0

	Count	Percentage
Machine failure
0	9661	96.61
1	339	3.39

	min	max	negative_values
Air temperature	295.3	304.5	0
Process temperature	305.7	313.8	0
Rotational speed	1168.0	2886.0	0
Torque	3.8	76.6	0
Tool wear	0.0	253.0	0

Rows where Process temperature < Air temperature: 0

Data-quality interpretation

The checks above should confirm:

• no missing sensor or target values
• no physically impossible negative values in the numeric operating columns
• no process-temperature values below air temperature
• a strongly imbalanced target distribution
• any exact duplicates requiring investigation before modeling

Because this dataset is synthetic, these checks primarily protect against loading errors or future source changes.

4. Exploratory Analysis

Class-conditional distributions are normalized independently so that the minority failure class remains visible.

Product-type failure rate

Type L has the lowest documented OSF threshold and is therefore easier to push into overstrain failure under otherwise similar operating conditions. Type H has the highest threshold.

	Failure rate (%)
Type
L	3.917
M	2.769
H	2.094

Domain-informed interactions

The following views are tied to documented failure mechanisms rather than arbitrary pairwise plots:

• torque versus rotational speed for power-related behavior
• mechanical-power estimate for PWF
• process-minus-air temperature gap versus speed for HDF

EDA Findings

Torque and Rotational speed show the clearest separation between the classes. Failures generally occur at higher torque and lower rotational speed, although a smaller failure group also appears at very low torque and unusually high speed. This indicates that failures are associated with extreme operating combinations rather than a single linear pattern.

The estimated-power plot supports this observation: failures are concentrated below the documented 3,500 W boundary and above 9,000 W, motivating the engineered Power feature. Tool wear is also higher for failures, with a visible concentration around 190–220 minutes, supporting both wear-related effects and the Torque x Tool wear interaction.

Air and process temperatures individually show substantial class overlap. However, failures are visible where the process-to-air temperature difference is below approximately 8.6 K and rotational speed is below 1,380 rpm. This motivates the engineered Temperature difference feature.

Failure rates also differ by product type: Type L has the highest observed rate (3.92%), followed by M (2.77%) and H (2.09%).

Overall, the EDA supports multivariate modeling because no single feature fully separates failures from non-failures. These findings are descriptive and specific to this synthetic benchmark.

5. Train/Test Split and Feature Engineering

The holdout test set is created before feature engineering and is not used for model selection, hyperparameter tuning, calibration analysis or threshold selection.

	Training	Test
Machine failure
0	0.9661	0.966
1	0.0339	0.034

Engineered features

Three domain-informed interactions are added:

• Power = torque × angular velocity
• Temperature difference = process temperature − air temperature
• Torque x Tool wear = torque × tool wear

These interactions reflect the documented PWF, HDF and OSF mechanisms while remaining valid transformations of the permitted input features.

	Power	Temperature difference	Torque x Tool wear
count	8000.000	8000.000	8000.000
mean	6282.618	10.001	4302.684
std	1072.419	1.000	2820.177
min	1148.441	7.600	0.000
25%	5559.606	9.300	1953.325
50%	6272.294	9.800	3998.700
75%	7013.867	11.000	6265.050
max	10469.923	12.100	16497.000

Raw-versus-engineered ablation

The same XGBoost configuration is evaluated on identical folds with and without the engineered features. Average Precision is the primary metric. F2 is included because missed failures are costly.

	Feature set	CV Average Precision	CV Average Precision Std	CV F2
0	Raw features only	0.7677	0.0336	0.6543
1	Raw + engineered features	0.8651	0.0299	0.8068

	Engineered minus raw AP by fold
0	0.0925
1	0.0479
2	0.1320
3	0.1106
4	0.1037

Mean paired AP difference: 0.0973

Interpret the paired fold differences rather than treating one model's fold standard deviation as a formal significance threshold. Retain the engineered features when they improve performance consistently or provide a defensible domain representation without materially degrading validation results.

6. Experimental Design

Training-set scale_pos_weight: 28.520
Candidate models: ['Dummy Baseline', 'Logistic Regression', 'Balanced Logistic Regression', 'L1 Logistic Regression', 'Decision Tree', 'Random Forest', 'XGBoost', 'Balanced XGBoost']

Evaluation policy

• Primary model-ranking metric: cross-validated Average Precision
• Secondary metrics: ROC-AUC, precision, recall, F1, and F2
• Threshold rule: maximize precision while maintaining recall at or above MIN_RECALL
• Final reporting: Average Precision, ROC-AUC, precision, recall, F1, F2, balanced accuracy, confusion counts, and bootstrap uncertainty

The test set remains untouched until the final evaluation section.

7. Baseline Model Comparison

	Model	CV Average Precision	CV Average Precision Std	CV ROC-AUC	CV Precision	CV Recall	CV F1	CV F2
0	Random Forest	0.8931	0.0244	0.9795	0.9002	0.8302	0.8633	0.8431
1	XGBoost	0.8651	0.0299	0.9790	0.9039	0.7861	0.8403	0.8068
2	Balanced XGBoost	0.8533	0.0387	0.9797	0.7688	0.8412	0.8026	0.8251
3	Decision Tree	0.7919	0.0589	0.9168	0.4010	0.8857	0.5517	0.7127
4	Logistic Regression	0.5000	0.0312	0.9142	0.7743	0.2510	0.3784	0.2901
5	Balanced Logistic Regression	0.4587	0.0416	0.9233	0.1621	0.8415	0.2716	0.4571
6	L1 Logistic Regression	0.4506	0.0413	0.9239	0.1601	0.8378	0.2688	0.4534
7	Dummy Baseline	0.0339	0.0003	0.5000	0.0000	0.0000	0.0000	0.0000

Select tuning candidates using the observed cross-validation ranking rather than expected model behavior. This repository tunes Random Forest and XGBoost because they are the designated nonlinear candidates. The results should explicitly state whether either actually outperformed the simpler baselines.

8. Hyperparameter Tuning

Best Random Forest parameters:
{'classifier__n_estimators': 400, 'classifier__min_samples_split': 2, 'classifier__min_samples_leaf': 1, 'classifier__max_features': 'log2', 'classifier__max_depth': 15}
Best Random Forest CV Average Precision: 0.8937

Fitting 5 folds for each of 30 candidates, totalling 150 fits

Best XGBoost parameters:
{'classifier__subsample': 1.0, 'classifier__reg_lambda': 1, 'classifier__reg_alpha': 0.5, 'classifier__n_estimators': 350, 'classifier__min_child_weight': 3, 'classifier__max_depth': 5, 'classifier__learning_rate': 0.01, 'classifier__gamma': 0.5, 'classifier__colsample_bytree': 0.8}
Best XGBoost CV Average Precision: 0.8812

	Model	Untuned CV Average Precision	Tuned CV Average Precision	Improvement
0	Random Forest	0.8931	0.8937	0.0007
1	XGBoost	0.8651	0.8812	0.0161

A negative tuning improvement is a legitimate result. It means the searched configuration did not outperform the existing baseline under the selected cross-validation design.

Selected final model among tuned candidates: Tuned Random Forest

The selection above is based only on training-set cross-validation scores. The estimator returned by RandomizedSearchCV is refitted on the full training set and remains isolated from the test labels.

9. Threshold Selection, Calibration, and Operational Cost

Recall-constrained threshold: 0.5750

	Threshold	Precision	Recall
0	0.0000	0.0339	1.0000
1	0.0004	0.0740	0.9889
2	0.0004	0.0743	0.9889
3	0.0004	0.0747	0.9889
4	0.0005	0.0780	0.9889
5	0.0006	0.0782	0.9889
6	0.0006	0.0783	0.9889
7	0.0006	0.0784	0.9889
8	0.0008	0.0785	0.9889
9	0.0010	0.0787	0.9889

Calibration

Calibration is necessary when model outputs are interpreted as failure probabilities rather than only ranking scores.

	Mean predicted probability	Observed failure rate
0	0.0000	0.0007
1	0.0020	0.0028
2	0.0048	0.0000
3	0.0091	0.0055
4	0.0346	0.0051
5	0.3999	0.3229

Illustrative expected-cost analysis

The configured false-negative and false-positive costs are scenario assumptions, not measured business values. The cost-minimizing threshold is shown as a sensitivity analysis. The recall-constrained threshold remains the primary operating threshold unless stakeholders validate the cost inputs.

Cost-minimizing threshold under configured assumptions: 0.1051

	Threshold	False Negatives	False Positives	Expected Cost
575	0.105	14	435	135250
574	0.105	14	438	135700
573	0.105	14	439	135850
572	0.105	14	440	136000
571	0.104	14	441	136150
570	0.102	14	446	136900
569	0.102	14	447	137050
568	0.102	14	448	137200
567	0.102	14	449	137350
566	0.102	14	450	137500

10. Final Test Evaluation

The test set is first exposed in this section. Afterward it is used only for descriptive diagnostics of the frozen model; no feature, model, hyperparameter or threshold is changed.

	Model	Threshold	Accuracy	Balanced Accuracy	Precision	Recall	F1-score	F2-score	ROC-AUC	Average Precision
0	Tuned Random Forest	0.575	0.9915	0.8963	0.9474	0.7941	0.864	0.8207	0.9762	0.8764

	precision	recall	f1-score	support
No failure	0.9928	0.9984	0.9956	1932.0000
Failure	0.9474	0.7941	0.8640	68.0000
accuracy	0.9915	0.9915	0.9915	0.9915
macro avg	0.9701	0.8963	0.9298	2000.0000
weighted avg	0.9913	0.9915	0.9911	2000.0000

Read the four artifacts above together, not independently. The metrics table and classification report provide point estimates at the fixed threshold, while the confusion matrix illustrates the underlying raw data. The ROC and Precision-Recall curves demonstrate performance across every threshold, not just the selected one. This makes them valuable for determining whether the chosen operating point is close to the optimal trade-off or if a significantly better point exists elsewhere on the curve.

As Average Precision, rather than ROC-AUC, is the model-selection metric used throughout this notebook (see Evaluation Policy, Section 7), give more weight to the Precision-Recall curve than to the ROC curve when judging how much headroom is left. ROC-AUC is reported for reference only, as under this level of class imbalance, it is the less informative of the two summaries.

Bootstrap confidence intervals

The test set contains relatively few failures, so point estimates can vary substantially. Percentile bootstrap intervals communicate this uncertainty more honestly than reporting metrics alone.

	Point estimate	Bootstrap mean	2.5th percentile	97.5th percentile
Average Precision	0.8764	0.8753	0.7979	0.9418
Recall	0.7941	0.7919	0.6875	0.8864
Precision	0.9474	0.9486	0.8809	1.0000
F1-score	0.8640	0.8622	0.7939	0.9219

11. Interpretability and Error Analysis

These analyses explain the already-frozen model. They must not feed back into model selection.

Permutation importance

Permutation importance measures the drop in held-out Average Precision when a single feature's values are randomly shuffled, holding everything else fixed. Because it is computed directly against a held-out performance metric rather than an internal split-count heuristic, it is treated as the primary importance signal in this notebook. The native (impurity- or gain-based) importance in the next subsection is reported for comparison but is secondary, see the note there for why the two can disagree.

	Feature	Importance Mean	Importance Std
3	Rotational speed	0.3897	0.0389
7	Temperature difference	0.2746	0.0258
8	Torque x Tool wear	0.1558	0.0098
6	Power	0.1135	0.0127
0	Type	0.0537	0.0201
5	Tool wear	0.0479	0.0078
4	Torque	0.0478	0.0131
1	Air temperature	0.0106	0.0110
2	Process temperature	-0.0105	0.0076

Native tree-model importance

Native importance is model-specific and can be biased toward continuous or high-cardinality features. It is therefore secondary to permutation importance.

	Feature	Importance
2	num__Rotational speed	0.2087
5	num__Power	0.1884
3	num__Torque	0.1766
4	num__Tool wear	0.1480
7	num__Torque x Tool wear	0.1136
6	num__Temperature difference	0.0900
0	num__Air temperature	0.0415
1	num__Process temperature	0.0251
9	cat__Type_M	0.0043
8	cat__Type_L	0.0039

Product-type slice performance

Always interpret slice metrics together with sample counts and failure prevalence. A high recall based on very few positive cases is unstable.

	Type	n	Failure rate	Recall	Precision	PR-AUC
0	H	214	0.0234	0.6000	0.7500	0.9029
1	L	1170	0.0325	0.7895	0.9677	0.8953
2	M	616	0.0406	0.8400	0.9545	0.8411

Failure-mode detection

A row may contain more than one active failure mode, so mode-specific case counts can overlap.

	Failure mode	Test cases	Caught (recall)
0	TWF	10	0.1000
1	HDF	29	0.9310
2	PWF	13	0.9231
3	OSF	16	1.0000
4	RNF	4	0.0000

False negatives and false positives

False negatives deserve particular attention because the operating objective prioritizes failure detection. TWF cases may be only partially predictable from tool wear, while RNF cases are not recoverable from the available operating inputs.

	Type	Air temperature	Process temperature	Rotational speed	Torque	Tool wear	Power	Temperature difference	Torque x Tool wear	Actual	Probability	Predicted	TWF	HDF	PWF	OSF	RNF
7884	L	300.8	312.4	1465	59.1	91	9066.793478	11.6	5378.1	1	0.527500	0	0	0	1	0	0
3695	L	302.2	311.3	1530	37.3	207	5976.251705	9.1	7721.1	1	0.523010	0	1	0	0	0	0
4307	L	301.4	309.9	1259	63.9	20	8424.714771	8.5	1278.0	1	0.517500	0	0	1	0	0	0
2864	H	300.6	309.4	1380	47.6	246	6878.831274	8.8	11709.6	1	0.444949	0	1	0	0	0	0
7087	L	300.6	310.3	1648	30.5	217	5263.633771	9.7	6618.5	1	0.371940	0	1	0	0	0	0
4034	L	302.0	310.8	1615	29.0	235	4904.549731	8.8	6815.0	1	0.323845	0	1	0	0	0	0
4778	H	303.6	312.2	1371	54.6	112	7838.964821	8.6	6115.2	1	0.317431	0	0	1	0	0	0
7849	M	300.3	311.7	1374	47.9	222	6892.088795	11.4	10633.8	1	0.117321	0	1	0	0	0	0
2941	M	300.7	309.6	1996	19.8	203	4138.608498	8.9	4019.4	1	0.086678	0	1	0	0	0	0
9015	L	297.2	308.1	1431	49.7	210	7447.742288	10.9	10437.0	1	0.079714	0	0	0	0	0	0

	Type	Air temperature	Process temperature	Rotational speed	Torque	Tool wear	Power	Temperature difference	Torque x Tool wear	Actual	Probability	Predicted
3794	M	302.2	310.8	1356	48.3	36	6858.599418	8.6	1738.8	0	0.934682	1
4231	L	302.2	310.8	1340	47.8	54	6707.509755	8.6	2581.2	0	0.912349	1
5988	H	300.7	310.8	1296	58.2	204	7898.717913	10.1	11872.8	0	0.587500	1

		Air temperature	Process temperature	Rotational speed	Torque	Tool wear	Power	Temperature difference	Torque x Tool wear
Actual	Predicted
0	0	300.00	310.10	1507.0	39.90	111.0	6235.988	9.8	3967.50
	1	302.20	310.80	1340.0	48.30	54.0	6858.599	8.6	2581.20
1	0	300.60	309.75	1448.0	40.05	205.0	6057.362	9.6	6716.75
	1	301.95	310.40	1349.5	55.35	149.5	7902.022	8.6	7811.30

12. Conclusion

Final model: Tuned Random Forest
Decision threshold: 0.575
Test Average Precision: 0.876 (95% CI [0.798, 0.942])
Test Recall: 0.794 (95% CI [0.687, 0.886])
Test Precision: 0.947

Final Interpretation

The Tuned Random Forest was selected as the final model with a cross-validated Average Precision of 0.8937. Tuning produced only a marginal improvement over the untuned Random Forest (0.8931 → 0.8937), while tuned XGBoost improved more noticeably (0.8651 → 0.8812) but remained weaker.

At the recall-constrained threshold of 0.575, the model achieved a test Average Precision of 0.876, recall of 0.794, precision of 0.947 and F2-score of approximately 0.821. It detected 54 of 68 failures, with 14 false negatives and only 3 false positives. Bootstrap intervals show uncertainty in Average Precision (0.798–0.942), recall (0.687–0.886) and precision (0.881–1.000).

Rotational speed was the strongest feature in both importance analyses. The engineered features Temperature difference, Torque x Tool wear and Power also contributed substantially, supporting the feature-engineering decisions.

Performance varied by product type. Type H had the weakest recall (0.600) and precision (0.750), although it contained only five failures. Type M achieved the lowest slice Average Precision (0.841), while Type L showed more balanced performance.

Failure-mode recall was strongest for OSF (1.000), HDF (0.931) and PWF (0.923). The model performed poorly on TWF, detecting only 1 of 10 cases and missed all 4 RNF cases. These small subgroup sizes make the estimates uncertain but they identify the main areas for further error analysis.

The cost-minimizing threshold of 0.105 was substantially lower than the recall-constrained threshold of 0.575, indicating that the assumed cost function favors a more aggressive strategy with more false alarms to reduce missed failures. The final threshold should therefore depend on the operational cost of inspections relative to undetected failures.

These findings are specific to the synthetic AI4I benchmark and do not demonstrate real-world deployment performance.

Limitations

• Synthetic benchmark: results do not establish production performance.
• Random stratified holdout: neighboring sequential operating states may appear across train and test; a chronological holdout should be added as a robustness experiment.
• Limited positive test cases: failure-class estimates have material uncertainty.
• Threshold-selection optimism: hyperparameters and the threshold are selected from the same overall training set, although threshold predictions are out of fold. A dedicated validation split or nested CV would provide stricter separation.
• Illustrative costs: false-positive and false-negative costs must be validated by domain stakeholders.
• Calibration: poor calibration would require a calibrated classifier before treating scores as literal probabilities.
• Failure-rule recoverability: because several synthetic failure rules are deterministic functions of the inputs, this benchmark partly measures recovery of known generation mechanisms.