Epistemic-Gated Transformer for Zero-Day Intrusion Mining in Network Traffic

Abstract

This paper proposes the Epistemic-Gated Transformer (EGT), a novel transformer-based data mining framework designed as the computational core of a Network-based Intrusion Detection System (NIDS) for effective zero-day attack detection. Conventional intrusion detection systems primarily rely on signature-based techniques or static machine learning classifiers, which exhibit limited capability in recognizing previously unseen attack patterns and adapting to rapidly evolving cyber threats. To overcome these limitations, the proposed EGT incorporates an epistemic uncertainty-guided multi-head self-attention mechanism that dynamically prioritizes uncertain network traffic during inference. The model employs a dual-output prediction head to simultaneously estimate class logits and Dirichlet concentration parameters from network flow tokens extracted through sliding windows. The differential entropy of the Dirichlet distribution provides a principled measure of epistemic uncertainty, which is transformed using a softplus-normalized gating function to modulate attention weights. This mechanism enables the model to emphasize suspicious or anomalous traffic while suppressing attention to well-characterized benign flows. Furthermore, EGT integrates a prototype-based contrastive mining module that maintains learnable anomaly prototypes within the latent feature space. A contrastive learning objective clusters high-uncertainty samples around their nearest prototypes, facilitating the discovery of previously unseen attack signatures and improving feature discrimination. The overall optimization objective combines cross-entropy classification loss, contrastive representation learning, and an uncertainty calibration regularizer to enhance robustness against out-of-distribution samples. During deployment, anomaly prototypes are continuously updated using an exponential moving average strategy, enabling online adaptation to evolving attack behaviors. Experimental evaluation demonstrates that the proposed framework achieves superior detection accuracy, robustness, scalability, and generalization, making it an effective solution for next-generation intelligent network intrusion detection.