Dual-Layer RSA–Paillier Encryption: Design and Evaluation Against Wiener Attacks
Received: 11 March 2026 | Revised: 3 April 2026, 24 April 2026, and 4 May 2026 | Accepted: 8 May 2026 | Online: 6 June 2026
Corresponding author: Dian Rachmawati
Abstract
Standard RSA encryption is vulnerable to Wiener's continued fraction attack when the private exponent is small, and its deterministic nature limits semantic security. This paper proposes a dual-layer encryption framework that combines RSA with the Paillier probabilistic cryptosystem to address these limitations. The scheme independently encrypts each plaintext under both RSA and Paillier using separate key pairs, transmits the ciphertext pair, and accepts the message only when both decrypted values agree. Because RSA relies on the Integer Factorization Problem (IFP) whereas Paillier relies on the Decisional Composite Residuosity Assumption (DCRA), the two layers provide orthogonal security guarantees: breaking the dual-layer scheme requires compromising both the IFP-based RSA layer and the DCRA-based Paillier layer. A formal adversarial model with a reduction-based security analysis demonstrates that the scheme achieves Indistinguishability under Chosen-Plaintext Attack (IND-CPA) semantic security by inheritance from Paillier, but not Indistinguishability under Adaptive Chosen-Ciphertext Attack (IND-CCA2). It also provides improved resilience against Wiener's attack, chosen-ciphertext exploits, and side-channel leakage compared with standalone RSA. Experimental evaluation using a Python implementation with key sizes of 1,024–4,096 bits shows that dual-layer encryption with a 2,048-bit modulus completes in approximately 98.19 ms, decryption in 125.04 ms, and the ciphertext expansion factor is 3× relative to RSA alone. Comparative analysis against RSA-only, RSA–Optimal Asymmetric Encryption Padding (RSA–OAEP), Paillier-only, and recent hybrid frameworks demonstrates that the proposed scheme provides a practical security--performance trade-off suitable for high-assurance applications where resilience against key-recovery attacks is a primary requirement.
Keywords:
RSA cryptosystem, Paillier cryptosystem, dual-layer encryption, hybrid cryptosystem, Wiener attack, continued fractions, homomorphic encryption, semantic securityReferences
W. Diffie and M. Hellman, "New directions in cryptography," IEEE Transactions on Information Theory, vol. 22, no. 6, pp. 644–654, Nov. 1976.
R. L. Rivest, A. Shamir, and L. Adleman, "A method for obtaining digital signatures and public-key cryptosystems," Communications of the ACM, vol. 21, no. 2, pp. 120–126, Feb. 1978.
M. J. Wiener, "Cryptanalysis of short RSA secret exponents," IEEE Transactions on Information Theory, vol. 36, no. 3, pp. 553–558, May 1990.
A. Dujella, "Continued fractions and RSA with small secret exponent," Tatra Mountains Mathematical Publications, vol. 29, no. 3, pp. 101–112, 2004.
A. Nitaj, "Another Generalization of Wiener’s Attack on RSA," in First International Conference on Cryptology in Africa, Casablanca, Morocco, 2008, pp. 174–190.
D. Boneh and G. Durfee, "Cryptanalysis of RSA with Private Key d Less than N0.292," in International Conference on the Theory and Application of Cryptographic Techniques, Prague, Czech Republic, 1999, pp. 1–11.
D. Bleichenbacher, "Chosen ciphertext attacks against protocols based on the RSA encryption standard PKCS #1," in 18th Annual International Cryptology Conference, Santa Barbara, CA, USA, 1998, pp. 1–12.
P. Kocher, J. Jaffe, and B. Jun, "Differential Power Analysis," in 19th Annual International Cryptology Conference, Santa Barbara, CA, USA, 1999, pp. 388–397.
P. Paillier, "Public-Key Cryptosystems Based on Composite Degree Residuosity Classes," in International Conference on the Theory and Application of Cryptographic Techniques, Prague, Czech Republic, 1999, pp. 223–238.
C. Marcolla, V. Sucasas, M. Manzano, R. Bassoli, F. H. P. Fitzek, and N. Aaraj, "Survey on Fully Homomorphic Encryption, Theory, and Applications," Proceedings of the IEEE, vol. 110, no. 10, pp. 1572–1609, Oct. 2022.
T. Elgamal, "A public key cryptosystem and a signature scheme based on discrete logarithms," IEEE Transactions on Information Theory, vol. 31, no. 4, pp. 469–472, July 1985.
N. R. D. P. Astuti, D. P. Setiawan, and D. C. Hakika, "Comparative Study of ElGamal and LUC Algorithm in Cryptographic Key Generation," Asean Engineering Journal, vol. 13, no. 4, pp. 61–68, Dec. 2023.
A. P. U. Siahaan, E. Elviwani, and B. Oktaviana, "Comparative Analysis of RSA and ElGamal Cryptographic Public-key Algorithms," in Joint Workshop KO2PI and The 1st International Conference on Advance & Scientific Innovation, Medan, Indonesia, 2018, pp. 163–172.
E. S. I. Harba, "Secure Data Encryption Through a Combination of AES, RSA and HMAC," Engineering, Technology & Applied Science Research, vol. 7, no. 4, pp. 1781–1785, Aug. 2017.
S. Bin-Faisal, D. Nandi, and M. Rahman, "Dual Layer Encryption for IoT based Vehicle Systems over 5G Communication," International Journal of Information Technology and Computer Science, vol. 14, no. 2, pp. 17–30.
A. Zabian, S. Mrayyen, A. M. Jonan, T. Al-Shaikh, and M. G. Al-Khaiyat, "Multi-layer encryption algorithm for data integrity in cloud computing," in Neural Networks, Machine Learning, and Image Processing, 1st. ed., M. Sahni, R. Sahni, and J. M. Merigo, Eds. Boca Raton, FL, USA: CRC Press, 2022, pp. 101–114.
M. N. Jeyakumar and J. Samraj, "Secure medical sensor monitoring framework using novel hybrid encryption algorithm driven by internet of things," Measurement: Sensors, vol. 33, June 2024, Art. no. 101122.
R. S. Kanakasabapathi and J. E. Judith, "An intelligent hybrid encryption framework for cloud systems in cybernetics using ISSO and Paillier cryptosystem," International Journal of Machine Learning and Cybernetics, vol. 16, no. 12, pp. 10541–10567, Dec. 2025.
B. Seth et al., "Secure Cloud Data Storage System Using Hybrid Paillier–Blowfish Algorithm," Computers, Materials & Continua, vol. 67, no. 1, pp. 779–798, Jan. 2021.
K. K. Almuzaini, A. K. Sinhal, R. Ranjan, V. Goel, R. Shrivastava, and Awal Halifa, "Key Aggregation Cryptosystem and Double Encryption Method for Cloud-Based Intelligent Machine Learning Techniques-Based Health Monitoring Systems," Computational Intelligence and Neuroscience, vol. 2022, no. 1, Apr. 2022, Art. no. 3767912.
A. Mishra, T. S. Jabar, Y. I. Alzoubi, and K. N. Mishra, "Enhancing privacy-preserving mechanisms in Cloud storage: A novel conceptual framework," Concurrency and Computation: Practice and Experience, vol. 35, no. 26, Nov. 2023, Art. no. e7831.
D.-T. Dam, T.-H. Tran, V.-P. Hoang, C.-K. Pham, and T.-T. Hoang, "A Survey of Post-Quantum Cryptography: Start of a New Race," Cryptography, vol. 7, no. 3, p. 40, Aug. 2023, Art. no. 40.
A. A.-R. El-Douh, S. F. Lu, A. Elkony, and A. S. Amein, "A Systematic Literature Review: The Taxonomy of Hybrid Cryptography Models," in Proceedings of the 2022 Future of Information and Communication Conference, San Francisco, CA, USA, 2022, pp. 714–721.
M. Bellare and P. Rogaway, "Optimal asymmetric encryption," in Workshop on the Theory and Application of Cryptographic Techniques, Perugia, Italy, 1994, pp. 92–111.
O. A. Qasim and S. Golshannavaz, "Enhancing data security using a multi-layer encryption system," International Journal of Electrical and Computer Engineering, vol. 15, no. 2, pp. 1961–1967, Apr. 2025.
D. Boneh and V. Shoup. "A Graduate Course in Applied Cryptography." Cryptobook. https://toc.cryptobook.us.
K. Moriarty, B. Kaliski, J. Jonsson, and A. Rusch, "PKCS #1: RSA Cryptography Specifications Version 2.2," Internet Engineering Task Force, Request for Comments RFC 8017, Nov. 2016.
Downloads
How to Cite
License
Copyright (c) 2026 Dian Rachmawati, Maya Silvi Lydia, Mohammad Andri Budiman, Romi Fadillah Rahmat
This work is licensed under a Creative Commons Attribution 4.0 International License.
Authors who publish with this journal agree to the following terms:
- Authors retain the copyright and grant the journal the right of first publication with the work simultaneously licensed under a Creative Commons Attribution License that allows others to share the work with an acknowledgement of the work's authorship and initial publication in this journal.
- Authors are able to enter into separate, additional contractual arrangements for the non-exclusive distribution of the journal's published version of the work (e.g., post it to an institutional repository or publish it in a book), with an acknowledgement of its initial publication in this journal.
- Authors are permitted and encouraged to post their work online (e.g., in institutional repositories or on their website) after its publication in ETASR with an acknowledgement of its initial publication in this journal.
